[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:26 dist-test-slave-2kcd krb5kdc[28133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:26 dist-test-slave-2kcd krb5kdc[28133](info): set up 2 sockets
Aug 10 21:09:26 dist-test-slave-2kcd krb5kdc[28133](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 21:09:28.271382 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.359s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 21:09:28.497633 28117 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.226s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 21:09:28.707870 28117 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.210s user 0.002s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 21:09:28.939222 28117 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.231s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/test-user.keytab.
W20260810 21:09:29.143080 28117 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.204s user 0.000s sys 0.003s
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:29.189800 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:34373
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:29.277384 28158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:29.277617 28158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:29.277658 28158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:29.278782 28158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:29.278849 28158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:29.278873 28158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:29.278903 28158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:29.278934 28158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:29.280593 28158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:29.281018 28158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:29.281327 28158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:29.288996 28176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.289029 28168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.289219 28169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:29.289486 28158 server_base.cc:1034] running on GCE node
I20260810 21:09:29.289829 28158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:29.290254 28158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:29.291425 28158 hybrid_clock.cc:648] HybridClock initialized: now 1786396169291404 us; error 86 us; skew 500 ppm
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:29.300050 28158 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:29.304739 28158 webserver.cc:460] Webserver started at http://127.27.117.126:43011/ using document root <none> and password file <none>
I20260810 21:09:29.305094 28158 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:29.305176 28158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:29.305415 28158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:29.307179 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "3f49ebb40b0a4a5a848859984fc9303b"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "0d1f711a9f8e2dfeedf3901c5098bb6b"
server_key_iv: "f87e15c19185885d2ceb79d131322802"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.307843 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "3f49ebb40b0a4a5a848859984fc9303b"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "0d1f711a9f8e2dfeedf3901c5098bb6b"
server_key_iv: "f87e15c19185885d2ceb79d131322802"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.309913 28158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260810 21:09:29.311081 28193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.311344 28158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:29.311474 28158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "3f49ebb40b0a4a5a848859984fc9303b"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "0d1f711a9f8e2dfeedf3901c5098bb6b"
server_key_iv: "f87e15c19185885d2ceb79d131322802"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.311569 28158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:29.322069 28158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:29.324143 28158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:29.324349 28158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:29.332302 28158 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:34373
I20260810 21:09:29.332355 28284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:34373 every 8 connection(s)
I20260810 21:09:29.333144 28158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:29.334652 28285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:29.334946 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28158
I20260810 21:09:29.335088 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:29.335862 28117 external_mini_cluster.cc:1428] Setting key 27355b30b5a407d4c7d9ba367ab29141
I20260810 21:09:29.338629 28285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b: Bootstrap starting.
I20260810 21:09:29.339884 28285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:29.340404 28285 log.cc:824] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:29.341620 28285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b: No bootstrap required, opened a new log
I20260810 21:09:29.344419 28285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49ebb40b0a4a5a848859984fc9303b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34373 } }
I20260810 21:09:29.344663 28285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:29.344755 28285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f49ebb40b0a4a5a848859984fc9303b, State: Initialized, Role: FOLLOWER
I20260810 21:09:29.345057 28285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49ebb40b0a4a5a848859984fc9303b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34373 } }
I20260810 21:09:29.345238 28285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:29.345278 28285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:29.345328 28285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:29.346951 28285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49ebb40b0a4a5a848859984fc9303b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34373 } }
I20260810 21:09:29.347160 28285 leader_election.cc:302] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f49ebb40b0a4a5a848859984fc9303b; no voters:
I20260810 21:09:29.347420 28285 leader_election.cc:288] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:29.347486 28292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:29.347908 28292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [term 1 LEADER]: Becoming Leader. State: Replica: 3f49ebb40b0a4a5a848859984fc9303b, State: Running, Role: LEADER
I20260810 21:09:29.348006 28285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:29.348089 28292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49ebb40b0a4a5a848859984fc9303b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34373 } }
I20260810 21:09:29.348959 28293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f49ebb40b0a4a5a848859984fc9303b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49ebb40b0a4a5a848859984fc9303b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34373 } } }
I20260810 21:09:29.348984 28294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f49ebb40b0a4a5a848859984fc9303b. Latest consensus state: current_term: 1 leader_uuid: "3f49ebb40b0a4a5a848859984fc9303b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f49ebb40b0a4a5a848859984fc9303b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34373 } } }
I20260810 21:09:29.349063 28293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:29.349081 28294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:29.350539 28314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:29.351831 28314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:29.357245 28314 catalog_manager.cc:1347] Generated new cluster ID: ccb0a08586a7443e9c940ff23784274c
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:29.357301 28314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:29.364434 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.337271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.27.117.126:34373)
0810 21:09:29.337782 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:29.337790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:29.337881 (+ 91us) server_negotiation.cc:407] Connection header received
0810 21:09:29.343389 (+ 5508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:29.343396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:29.343489 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:29.343607 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:29.346748 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.348641 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.350845 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.351692 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.351888 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.358237 (+ 6349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:29.358241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:29.358242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:29.358254 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:29.361448 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.361945 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.361961 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.361961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.362114 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.362419 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.362420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.362420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.362793 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:29.363172 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:29.363309 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 21:09:29.363546 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":155,"threads_started":1}
I20260810 21:09:29.374675 28314 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:29.375487 28314 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:29.382608 28314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f49ebb40b0a4a5a848859984fc9303b: Generated new TSK 0
I20260810 21:09:29.382795 28314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:29.419281 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34373
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:29.529103 28334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:29.529299 28334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:29.529326 28334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:29.530575 28334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:29.530619 28334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:29.530647 28334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:29.532341 28334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34373
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:29.532637 28334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:29.532948 28334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:29.539772 28342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.539794 28347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.539865 28358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:29.540284 28334 server_base.cc:1034] running on GCE node
I20260810 21:09:29.540643 28334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:29.541128 28334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:29.542459 28334 hybrid_clock.cc:648] HybridClock initialized: now 1786396169542429 us; error 29 us; skew 500 ppm
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:29.546746 28334 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:29.547262 28334 webserver.cc:460] Webserver started at http://127.27.117.65:33375/ using document root <none> and password file <none>
I20260810 21:09:29.547514 28334 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:29.547559 28334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:29.547717 28334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:29.549188 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "2cf8f6d2eaa541eaa935c3a2013a3529"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "06f0b1cb9271c901e8fbb085bc6ea83b"
server_key_iv: "534946885800aeb95d3fc8f6b010c548"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.549566 28334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "2cf8f6d2eaa541eaa935c3a2013a3529"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "06f0b1cb9271c901e8fbb085bc6ea83b"
server_key_iv: "534946885800aeb95d3fc8f6b010c548"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.551357 28334 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:29.552474 28370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.552676 28334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:29.552765 28334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "2cf8f6d2eaa541eaa935c3a2013a3529"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "06f0b1cb9271c901e8fbb085bc6ea83b"
server_key_iv: "534946885800aeb95d3fc8f6b010c548"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.552910 28334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:29.575464 28334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:29.578065 28334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:29.578258 28334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:29.578722 28334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:29.579124 28334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:29.579203 28334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.579227 28334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:29.579273 28334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.594581 28334 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:36451
I20260810 21:09:29.594694 28595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:36451 every 8 connection(s)
I20260810 21:09:29.594986 28334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:29.595595 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28334
I20260810 21:09:29.595691 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:29.595893 28117 external_mini_cluster.cc:1428] Setting key 2cda9be1b85be32bc2d19aaf96448211
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
I20260810 21:09:29.607039 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.595930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:54425 (local address 127.27.117.126:34373)
0810 21:09:29.595991 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:29.595993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:29.596459 (+ 466us) server_negotiation.cc:407] Connection header received
0810 21:09:29.597639 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:29.597639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:29.597649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:29.597697 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:29.599561 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.600052 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.601260 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.601503 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.601594 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.604016 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:29.604018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:29.604018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:29.604026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:29.605620 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.606034 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.606036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.606036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.606100 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.606369 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.606370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.606370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.606485 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:29.606576 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:29.606861 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 21:09:29.606948 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 21:09:29.607290 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.595901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:34373 (local address 127.27.117.65:54425)
0810 21:09:29.596290 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:29.596339 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:29.597527 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:29.597777 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:29.597782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:29.598917 (+ 1135us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:29.599457 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:29.599463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.600147 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.600148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:29.601158 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:29.601161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.601608 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.601609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:29.601802 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.602418 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:29.602426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:29.603920 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:29.605706 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:29.605714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:29.605716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:29.605941 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:29.606176 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:29.606177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:29.606177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:29.606291 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:29.606569 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:29.606577 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:29.606726 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 21:09:29.606837 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 21:09:29.606989 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":214,"threads_started":1}
I20260810 21:09:29.607961 28596 heartbeater.cc:347] Connected to a master server at 127.27.117.126:34373
I20260810 21:09:29.608053 28596 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:29.608192 28596 heartbeater.cc:511] Master 127.27.117.126:34373 requested a full tablet report, sending...
I20260810 21:09:29.609063 28221 ts_manager.cc:194] Registered new tserver with Master: 2cf8f6d2eaa541eaa935c3a2013a3529 (127.27.117.65:36451)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
I20260810 21:09:29.610014 28221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:54425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:29.625869 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34373
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:29.732951 28603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:29.733150 28603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:29.733180 28603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:29.734305 28603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:29.734344 28603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:29.734371 28603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:29.735986 28603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34373
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:29.736174 28603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:29.736356 28603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:29.742573 28610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.742559 28615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.742673 28611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:29.742874 28603 server_base.cc:1034] running on GCE node
I20260810 21:09:29.743090 28603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:29.743296 28603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:29.744473 28603 hybrid_clock.cc:648] HybridClock initialized: now 1786396169744459 us; error 31 us; skew 500 ppm
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:29.747607 28603 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:29.748136 28603 webserver.cc:460] Webserver started at http://127.27.117.66:41083/ using document root <none> and password file <none>
I20260810 21:09:29.748399 28603 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:29.748499 28603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:29.748670 28603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:29.749886 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "c1f2503ad1a845e599919c736b8f8af0"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "87916bf091a1ce8dce9254c59eee1ece"
server_key_iv: "0e98d9cf7294ec6d80390c86c9580ae3"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.750311 28603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "c1f2503ad1a845e599919c736b8f8af0"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "87916bf091a1ce8dce9254c59eee1ece"
server_key_iv: "0e98d9cf7294ec6d80390c86c9580ae3"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.751758 28603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:29.752635 28629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.752861 28603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:29.752969 28603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "c1f2503ad1a845e599919c736b8f8af0"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "87916bf091a1ce8dce9254c59eee1ece"
server_key_iv: "0e98d9cf7294ec6d80390c86c9580ae3"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.753052 28603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:29.767989 28603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:29.770324 28603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:29.770550 28603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:29.770956 28603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:29.771384 28603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:29.771446 28603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.771489 28603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:29.771538 28603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.787972 28603 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36265
I20260810 21:09:29.788010 28856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36265 every 8 connection(s)
I20260810 21:09:29.788408 28603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:29.791900 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28603
I20260810 21:09:29.792007 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:29.792158 28117 external_mini_cluster.cc:1428] Setting key adbb41dabb8be4a7e4b87eefb4c434e4
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:29.800611 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.789274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:34373 (local address 127.27.117.66:44415)
0810 21:09:29.789583 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:29.789641 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:29.790492 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:29.790792 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:29.790795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:29.791474 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:29.791784 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:29.791789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.792736 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.792737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:29.793780 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:29.793782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.794214 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.794215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:29.794388 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.795239 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:29.795250 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:29.797178 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:29.799238 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:29.799245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:29.799246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:29.799484 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:29.799726 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:29.799727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:29.799727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:29.799822 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:29.800125 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:29.800130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:29.800240 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 21:09:29.800322 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 21:09:29.800452 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":155,"threads_started":1}
I20260810 21:09:29.800658 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.789300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:44415 (local address 127.27.117.126:34373)
0810 21:09:29.789388 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:29.789390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:29.789768 (+ 378us) server_negotiation.cc:407] Connection header received
0810 21:09:29.790625 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:29.790627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:29.790642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:29.790716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:29.791902 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.792620 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.793886 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.794108 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.794204 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.797275 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:29.797277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:29.797277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:29.797287 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:29.799145 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.799564 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.799566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.799566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.799638 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.799889 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.799890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.799890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.800051 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:29.800139 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:29.800378 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 21:09:29.800473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:09:29.801262 28857 heartbeater.cc:347] Connected to a master server at 127.27.117.126:34373
I20260810 21:09:29.801388 28857 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:29.801589 28857 heartbeater.cc:511] Master 127.27.117.126:34373 requested a full tablet report, sending...
I20260810 21:09:29.802027 28221 ts_manager.cc:194] Registered new tserver with Master: c1f2503ad1a845e599919c736b8f8af0 (127.27.117.66:36265)
I20260810 21:09:29.802457 28221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:44415
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:29.823441 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34373
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:29.941520 28866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:29.941741 28866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:29.941784 28866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:29.942976 28866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:29.943042 28866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:29.943087 28866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:29.944664 28866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34373
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:29.944940 28866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:29.945169 28866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:29.950619 28874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.950764 28875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:29.950764 28881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:29.950964 28866 server_base.cc:1034] running on GCE node
I20260810 21:09:29.951181 28866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:29.951443 28866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:29.952590 28866 hybrid_clock.cc:648] HybridClock initialized: now 1786396169952571 us; error 36 us; skew 500 ppm
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:29.955667 28866 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:29.956169 28866 webserver.cc:460] Webserver started at http://127.27.117.67:38901/ using document root <none> and password file <none>
I20260810 21:09:29.956432 28866 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:29.956501 28866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:29.956633 28866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:29.957962 28866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "23cf89abc98d43a082c86315f2fbbe27"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "6a9bf7aba42bec6483959265247d9ecd"
server_key_iv: "317a3596da738955da31b31f896f1cc6"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.958387 28866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "23cf89abc98d43a082c86315f2fbbe27"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "6a9bf7aba42bec6483959265247d9ecd"
server_key_iv: "317a3596da738955da31b31f896f1cc6"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.959875 28866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:09:29.960664 28895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.960875 28866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:29.960963 28866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "23cf89abc98d43a082c86315f2fbbe27"
format_stamp: "Formatted at 2026-08-10 21:09:29 on dist-test-slave-2kcd"
server_key: "6a9bf7aba42bec6483959265247d9ecd"
server_key_iv: "317a3596da738955da31b31f896f1cc6"
server_key_version: "encryptionkey@0"
I20260810 21:09:29.961014 28866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:29.967854 28866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:29.970044 28866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:29.970242 28866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:29.970594 28866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:29.970937 28866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:29.970995 28866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.971035 28866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:29.971062 28866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:29.984989 28866 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:35935
I20260810 21:09:29.985105 29097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:35935 every 8 connection(s)
I20260810 21:09:29.985407 28866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:29.989319 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28866
I20260810 21:09:29.989388 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:29.989550 28117 external_mini_cluster.cc:1428] Setting key 40b1dd818e01c64ea9bfb84f0e57b4e7
Aug 10 21:09:29 dist-test-slave-2kcd krb5kdc[28133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396169, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:29.995445 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.986172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:39537 (local address 127.27.117.126:34373)
0810 21:09:29.986214 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:29.986216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:29.986510 (+ 294us) server_negotiation.cc:407] Connection header received
0810 21:09:29.987298 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:29.987300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:29.987313 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:29.987366 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:29.988377 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.988975 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.990401 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.990646 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.990710 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.992998 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:29.992999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:29.992999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:29.993006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:29.994404 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.994732 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.994733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.994734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.994782 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:29.994981 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:29.994982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:29.994982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:29.995076 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:29.995127 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:29.995323 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 21:09:29.995381 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:09:29.995707 29104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:29.986149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:34373 (local address 127.27.117.67:39537)
0810 21:09:29.986392 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:29.986424 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:29.987185 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:29.987442 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:29.987446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:29.988038 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:29.988258 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:29.988262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.989073 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.989074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:29.990305 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:29.990308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:29.990728 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:29.990730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:29.990899 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:29.991369 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:29.991377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:29.992919 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:29.994470 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:29.994475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:29.994476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:29.994668 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:29.994839 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:29.994839 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:29.994839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:29.994928 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:29.995130 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:29.995133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:29.995230 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 21:09:29.995310 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 21:09:29.995421 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":151,"threads_started":1}
I20260810 21:09:29.996261 29098 heartbeater.cc:347] Connected to a master server at 127.27.117.126:34373
I20260810 21:09:29.996361 29098 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:29.996531 29098 heartbeater.cc:511] Master 127.27.117.126:34373 requested a full tablet report, sending...
I20260810 21:09:29.996920 28221 ts_manager.cc:194] Registered new tserver with Master: 23cf89abc98d43a082c86315f2fbbe27 (127.27.117.67:35935)
I20260810 21:09:29.997241 28221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:39537
I20260810 21:09:30.001401 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[28133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396170, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:30.027963 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:30.019716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.27.117.65:36451)
0810 21:09:30.019965 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:30.019969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:30.019979 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:30.020004 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:30.020006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:30.020020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:30.020201 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:30.020751 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:30.021572 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:30.022255 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:30.022651 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:30.022709 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:30.024818 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:30.024829 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:30.024830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:30.024836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:30.026706 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:30.027053 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:30.027056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:30.027057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:30.027214 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:30.027391 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:30.027392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:30.027392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:30.027524 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:30.027624 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:30.027687 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 21:09:30.027753 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260810 21:09:30.028447 28424 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34254
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[28133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[28133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:30.052569 29110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:30.045333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.27.117.65:36451)
0810 21:09:30.045417 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:30.045419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:30.045520 (+ 101us) server_negotiation.cc:407] Connection header received
0810 21:09:30.045598 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:30.045599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:30.045611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:30.045683 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:30.046221 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:30.046692 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:30.047427 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:30.047678 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:30.047763 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:30.050184 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:30.050186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:30.050186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:30.050193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:30.051404 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:30.051777 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:30.051779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:30.051779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:30.051856 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:30.052056 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:30.052057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:30.052057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:30.052163 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:30.052288 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:30.052390 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 21:09:30.052468 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 21:09:30.053306 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28334
I20260810 21:09:30.075564 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28603
I20260810 21:09:30.088543 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28866
I20260810 21:09:30.095187 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28158
2026-08-10T21:09:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3220 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[29132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[29132](info): set up 2 sockets
Aug 10 21:09:30 dist-test-slave-2kcd krb5kdc[29132](info): commencing operation
krb5kdc: starting...
W20260810 21:09:31.387991 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:31.505530 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:44765
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:44765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:31.585728 29158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:31.585925 29158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:31.585947 29158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:31.586997 29158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:31.587028 29158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:31.587041 29158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:31.587054 29158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:31.587064 29158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:31.588433 29158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:44765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:44765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:31.588625 29158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:31.588819 29158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:31.593613 29174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:31.593719 29158 server_base.cc:1034] running on GCE node
W20260810 21:09:31.593636 29177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:31.593840 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:31.594097 29158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:31.594324 29158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:31.595463 29158 hybrid_clock.cc:648] HybridClock initialized: now 1786396171595442 us; error 37 us; skew 500 ppm
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:31.597921 29158 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:31.598374 29158 webserver.cc:460] Webserver started at http://127.27.117.126:43097/ using document root <none> and password file <none>
I20260810 21:09:31.598598 29158 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:31.598665 29158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:31.598819 29158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:31.599849 29158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a8fb7ff7a7b74c98aab2c19570efd90f"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "27f5709ab0d6cd25cd3a4a6f5cd9d97c"
server_key_iv: "ad7b9f92260d1a17ceb86b7d27df3a1d"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.600224 29158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a8fb7ff7a7b74c98aab2c19570efd90f"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "27f5709ab0d6cd25cd3a4a6f5cd9d97c"
server_key_iv: "ad7b9f92260d1a17ceb86b7d27df3a1d"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.601614 29158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:31.602380 29190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.602555 29158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:31.602667 29158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a8fb7ff7a7b74c98aab2c19570efd90f"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "27f5709ab0d6cd25cd3a4a6f5cd9d97c"
server_key_iv: "ad7b9f92260d1a17ceb86b7d27df3a1d"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.602741 29158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:31.625955 29158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:31.627751 29158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:31.627919 29158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:31.632246 29158 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:44765
I20260810 21:09:31.632282 29283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:44765 every 8 connection(s)
I20260810 21:09:31.632639 29158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:31.633534 29284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:31.635852 29284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f: Bootstrap starting.
I20260810 21:09:31.636392 29284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:31.636628 29284 log.cc:824] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:31.637367 29284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f: No bootstrap required, opened a new log
I20260810 21:09:31.638835 29284 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44765 } }
I20260810 21:09:31.638963 29284 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:31.638985 29284 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fb7ff7a7b74c98aab2c19570efd90f, State: Initialized, Role: FOLLOWER
I20260810 21:09:31.639093 29284 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44765 } }
I20260810 21:09:31.639185 29284 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:31.639214 29284 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:31.639257 29284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:31.639828 29284 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44765 } }
I20260810 21:09:31.639941 29284 leader_election.cc:302] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8fb7ff7a7b74c98aab2c19570efd90f; no voters:
I20260810 21:09:31.640118 29284 leader_election.cc:288] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:31.640121 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29158
I20260810 21:09:31.640143 29289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:31.640210 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:31.640259 29289 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [term 1 LEADER]: Becoming Leader. State: Replica: a8fb7ff7a7b74c98aab2c19570efd90f, State: Running, Role: LEADER
I20260810 21:09:31.640334 29289 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44765 } }
I20260810 21:09:31.640370 28117 external_mini_cluster.cc:1428] Setting key 0ddf5ab09afce70fe710604576f3f356
I20260810 21:09:31.640369 29284 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:31.640682 29289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8fb7ff7a7b74c98aab2c19570efd90f. Latest consensus state: current_term: 1 leader_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44765 } } }
I20260810 21:09:31.640750 29289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:31.641106 29292 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8fb7ff7a7b74c98aab2c19570efd90f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44765 } } }
I20260810 21:09:31.641194 29292 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:31.641287 29299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:31.641816 29299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:31.644182 29299 catalog_manager.cc:1347] Generated new cluster ID: 2dac30a767074207ba5321623cb1e211
I20260810 21:09:31.644236 29299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:31.652544 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:31.640993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.27.117.126:44765)
0810 21:09:31.641552 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:31.641562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:31.641596 (+ 34us) server_negotiation.cc:407] Connection header received
0810 21:09:31.641711 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:31.641717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:31.641782 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:31.641929 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:31.642681 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.643737 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.644771 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.645432 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.645558 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:31.648224 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:31.648228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:31.648229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:31.648239 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:31.650932 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:31.651344 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:31.651346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:31.651347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:31.651457 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:31.651679 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:31.651679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:31.651679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:31.651957 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:31.652101 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:31.652207 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 21:09:31.652330 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":103,"threads_started":1}
I20260810 21:09:31.657500 29299 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:31.658085 29299 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:31.665035 29299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8fb7ff7a7b74c98aab2c19570efd90f: Generated new TSK 0
I20260810 21:09:31.665208 29299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:31.701113 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44765
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:31.792482 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:31.792672 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:31.792697 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:31.793846 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:31.793893 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:31.793951 29331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:31.795475 29331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44765
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:31.795724 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:31.795974 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:31.801160 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:31.801167 29331 server_base.cc:1034] running on GCE node
W20260810 21:09:31.801151 29349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:31.801136 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:31.801571 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:31.801844 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:31.802997 29331 hybrid_clock.cc:648] HybridClock initialized: now 1786396171802986 us; error 33 us; skew 500 ppm
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:31.805576 29331 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:31.805995 29331 webserver.cc:460] Webserver started at http://127.27.117.65:40969/ using document root <none> and password file <none>
I20260810 21:09:31.806245 29331 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:31.806291 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:31.806465 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:31.807574 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "74bbae9767df4b78bf1b825e5192e28d"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "ead85a96575b802547a0293b5286af75"
server_key_iv: "82a4b884966c52863efc8aa7ea77cc8f"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.807932 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "74bbae9767df4b78bf1b825e5192e28d"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "ead85a96575b802547a0293b5286af75"
server_key_iv: "82a4b884966c52863efc8aa7ea77cc8f"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.809401 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:09:31.810139 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.810328 29331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:31.810421 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "74bbae9767df4b78bf1b825e5192e28d"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "ead85a96575b802547a0293b5286af75"
server_key_iv: "82a4b884966c52863efc8aa7ea77cc8f"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.810497 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:31.817091 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:31.818935 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:31.819103 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:31.819334 29331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:31.819626 29331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:31.819661 29331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.819682 29331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:31.819726 29331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.826105 29331 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:44017
I20260810 21:09:31.826243 29566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:44017 every 8 connection(s)
I20260810 21:09:31.826493 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:31.826650 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29331
I20260810 21:09:31.826766 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:31.826960 28117 external_mini_cluster.cc:1428] Setting key c0f270bc7d71aa0f6d8a031178ac855f
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
I20260810 21:09:31.838521 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:31.827319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:33033 (local address 127.27.117.126:44765)
0810 21:09:31.827628 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:31.827631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:31.827641 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:31.829376 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:31.829377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:31.829387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:31.829426 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:31.830246 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.831069 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.832675 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.832972 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.833027 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:31.835515 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:31.835517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:31.835518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:31.835526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:31.837006 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:31.837430 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:31.837431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:31.837431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:31.837503 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:31.837768 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:31.837769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:31.837769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:31.837925 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:31.838022 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:31.838331 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 21:09:31.838425 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260810 21:09:31.838670 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:31.827311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:44765 (local address 127.27.117.65:33033)
0810 21:09:31.827499 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:31.827546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:31.829267 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:31.829498 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:31.829502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:31.829909 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:31.830152 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:31.830158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.831171 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.831173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:31.832584 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:31.832588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.833064 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.833066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:31.833236 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:31.833734 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:31.833744 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:31.835415 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:31.837093 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:31.837101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:31.837103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:31.837366 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:31.837580 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:31.837581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:31.837581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:31.837692 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:31.838013 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:31.838021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:31.838164 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 21:09:31.838291 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 21:09:31.838447 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260810 21:09:31.839212 29568 heartbeater.cc:347] Connected to a master server at 127.27.117.126:44765
I20260810 21:09:31.839295 29568 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:31.839433 29568 heartbeater.cc:511] Master 127.27.117.126:44765 requested a full tablet report, sending...
I20260810 21:09:31.840070 29215 ts_manager.cc:194] Registered new tserver with Master: 74bbae9767df4b78bf1b825e5192e28d (127.27.117.65:44017)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
I20260810 21:09:31.841035 29215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:33033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:31.855625 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44765
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:31.937341 29573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:31.937526 29573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:31.937551 29573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:31.938606 29573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:31.938639 29573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:31.938665 29573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:31.940224 29573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44765
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:31.940407 29573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:31.940574 29573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:31.945636 29583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:31.945612 29584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:31.945595 29590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:31.945927 29573 server_base.cc:1034] running on GCE node
I20260810 21:09:31.946136 29573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:31.946326 29573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:31.947503 29573 hybrid_clock.cc:648] HybridClock initialized: now 1786396171947489 us; error 29 us; skew 500 ppm
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:31.950066 29573 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:31.950541 29573 webserver.cc:460] Webserver started at http://127.27.117.66:41763/ using document root <none> and password file <none>
I20260810 21:09:31.950785 29573 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:31.950831 29573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:31.951007 29573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:31.952016 29573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "53099f562d214f5ba5b57d626a70039c"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "0685168c73cfff8986b83ba7de21f10e"
server_key_iv: "712e28ae905ac686665fdf0d873628cb"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.952368 29573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "53099f562d214f5ba5b57d626a70039c"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "0685168c73cfff8986b83ba7de21f10e"
server_key_iv: "712e28ae905ac686665fdf0d873628cb"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.953778 29573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:31.954524 29607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.954716 29573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:31.954811 29573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "53099f562d214f5ba5b57d626a70039c"
format_stamp: "Formatted at 2026-08-10 21:09:31 on dist-test-slave-2kcd"
server_key: "0685168c73cfff8986b83ba7de21f10e"
server_key_iv: "712e28ae905ac686665fdf0d873628cb"
server_key_version: "encryptionkey@0"
I20260810 21:09:31.954890 29573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:31.976536 29573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:31.978379 29573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:31.978545 29573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:31.978785 29573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:31.979126 29573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:31.979161 29573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.979183 29573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:31.979233 29573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:31.985508 29573 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:34953
I20260810 21:09:31.985563 29812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:34953 every 8 connection(s)
I20260810 21:09:31.985865 29573 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:31.990619 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29573
I20260810 21:09:31.990691 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:31.990854 28117 external_mini_cluster.cc:1428] Setting key 2caf3ca659e5d5a3ac92118df40bdb24
Aug 10 21:09:31 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:31.995488 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:31.986591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:48677 (local address 127.27.117.126:44765)
0810 21:09:31.986656 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:31.986659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:31.986843 (+ 184us) server_negotiation.cc:407] Connection header received
0810 21:09:31.987808 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:31.987810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:31.987824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:31.987881 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:31.988492 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.989096 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.990150 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.990360 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.990447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:31.992607 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:31.992609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:31.992609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:31.992616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:31.994103 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:31.994531 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:31.994532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:31.994533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:31.994595 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:31.994854 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:31.994855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:31.994855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:31.994969 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:31.995050 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:31.995317 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 21:09:31.995396 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:09:31.995702 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:31.986567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:44765 (local address 127.27.117.66:48677)
0810 21:09:31.986724 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:31.986758 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:31.987693 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:31.987948 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:31.987952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:31.988214 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:31.988394 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:31.988398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.989191 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.989192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:31.989999 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:31.990001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:31.990460 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:31.990461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:31.990601 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:31.991167 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:31.991175 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:31.992504 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:31.994205 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:31.994213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:31.994215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:31.994439 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:31.994669 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:31.994670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:31.994670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:31.994780 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:31.995044 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:31.995052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:31.995188 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 21:09:31.995295 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 21:09:31.995451 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260810 21:09:31.996165 29813 heartbeater.cc:347] Connected to a master server at 127.27.117.126:44765
I20260810 21:09:31.996263 29813 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:31.996435 29813 heartbeater.cc:511] Master 127.27.117.126:44765 requested a full tablet report, sending...
I20260810 21:09:31.996852 29215 ts_manager.cc:194] Registered new tserver with Master: 53099f562d214f5ba5b57d626a70039c (127.27.117.66:34953)
I20260810 21:09:31.997413 29215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:48677
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:32.016705 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44765
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:32.097913 29822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:32.098105 29822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:32.098129 29822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:32.099192 29822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:32.099227 29822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:32.099253 29822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:32.100657 29822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44765
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:32.100957 29822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:32.101154 29822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:32.106191 29836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:32.106282 29828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:32.106364 29822 server_base.cc:1034] running on GCE node
W20260810 21:09:32.106240 29829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:32.106652 29822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:32.106854 29822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:32.108031 29822 hybrid_clock.cc:648] HybridClock initialized: now 1786396172108016 us; error 39 us; skew 500 ppm
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:32.110579 29822 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:32.111084 29822 webserver.cc:460] Webserver started at http://127.27.117.67:32857/ using document root <none> and password file <none>
I20260810 21:09:32.111326 29822 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:32.111397 29822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:32.111550 29822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:32.112561 29822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "4a9023a3569446c89076f5a0f6751a5b"
format_stamp: "Formatted at 2026-08-10 21:09:32 on dist-test-slave-2kcd"
server_key: "84c1a3f982f3c6f80e40a2ac74cbcc39"
server_key_iv: "3d58ca24294aecaadaac760af23663cc"
server_key_version: "encryptionkey@0"
I20260810 21:09:32.112952 29822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "4a9023a3569446c89076f5a0f6751a5b"
format_stamp: "Formatted at 2026-08-10 21:09:32 on dist-test-slave-2kcd"
server_key: "84c1a3f982f3c6f80e40a2ac74cbcc39"
server_key_iv: "3d58ca24294aecaadaac760af23663cc"
server_key_version: "encryptionkey@0"
I20260810 21:09:32.114427 29822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:32.115152 29846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:32.115330 29822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:32.115437 29822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "4a9023a3569446c89076f5a0f6751a5b"
format_stamp: "Formatted at 2026-08-10 21:09:32 on dist-test-slave-2kcd"
server_key: "84c1a3f982f3c6f80e40a2ac74cbcc39"
server_key_iv: "3d58ca24294aecaadaac760af23663cc"
server_key_version: "encryptionkey@0"
I20260810 21:09:32.115520 29822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:32.132441 29822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:32.134227 29822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:32.134388 29822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:32.134616 29822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:32.134938 29822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:32.135000 29822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:32.135036 29822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:32.135078 29822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:32.142496 29822 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:37635
I20260810 21:09:32.142573 30061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:37635 every 8 connection(s)
I20260810 21:09:32.142895 29822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396172, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:32.151441 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29822
I20260810 21:09:32.151546 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:32.151746 28117 external_mini_cluster.cc:1428] Setting key aeeb89d3a8d9ecd2246a88865ee1e613
I20260810 21:09:32.152541 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.143705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:43979 (local address 127.27.117.126:44765)
0810 21:09:32.143755 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.143758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.144053 (+ 295us) server_negotiation.cc:407] Connection header received
0810 21:09:32.145209 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.145210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.145221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.145275 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:32.145831 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.146317 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.147251 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.147431 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.147480 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.149734 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:32.149735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:32.149735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:32.149742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:32.150938 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.151330 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.151331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.151331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.151384 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.151694 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.151695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.151695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.151828 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:32.151916 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.152349 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.152443 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:09:32.152853 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.143700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:44765 (local address 127.27.117.67:43979)
0810 21:09:32.143911 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:32.143948 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:32.145119 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:32.145325 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:32.145328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:32.145580 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:32.145763 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.145767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.146393 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.146393 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.147181 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.147183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.147500 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.147501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.147610 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.148128 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:32.148136 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:32.149648 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:32.151014 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:32.151020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:32.151022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:32.151251 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:32.151453 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:32.151454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:32.151454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:32.151621 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:32.152014 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:32.152022 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:32.152191 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 21:09:32.152312 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 21:09:32.152473 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260810 21:09:32.153290 30062 heartbeater.cc:347] Connected to a master server at 127.27.117.126:44765
I20260810 21:09:32.153391 30062 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:32.153564 30062 heartbeater.cc:511] Master 127.27.117.126:44765 requested a full tablet report, sending...
I20260810 21:09:32.153937 29215 ts_manager.cc:194] Registered new tserver with Master: 4a9023a3569446c89076f5a0f6751a5b (127.27.117.67:37635)
I20260810 21:09:32.154309 29215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:43979
I20260810 21:09:32.163556 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:09:32.171069 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.165494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.27.117.126:44765)
0810 21:09:32.165550 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.165552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.165702 (+ 150us) server_negotiation.cc:407] Connection header received
0810 21:09:32.165739 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.165740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.165748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.165787 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:32.166319 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.166786 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.167437 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.167617 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.167668 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.168760 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:32.168761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:32.168761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:32.168767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:32.170164 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.170467 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.170468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.170468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.170516 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.170679 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.170679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.170679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.170806 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:32.170877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.170957 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.171009 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:09:32.173599 29215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:32.175388 29215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:32.183400 30092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.178451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:37635 (local address 127.0.0.1:35400)
0810 21:09:32.178911 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:32.178922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:32.179150 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:32.179614 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:32.179617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:32.179619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:32.179675 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.179677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.181127 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.181128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.182125 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.182127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.183098 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.183099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.183174 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.183187 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 21:09:32.183268 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:09:32.183312 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":397,"threads_started":1}
I20260810 21:09:32.183600 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.178643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.27.117.65:44017)
0810 21:09:32.179280 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.179285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.179295 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:32.179371 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.179383 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.179413 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.179647 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:32.179947 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.180868 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.182268 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.182989 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.183365 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.183446 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.183474 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.183521 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":82,"threads_started":1}
I20260810 21:09:32.183434 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.178583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.27.117.67:37635)
0810 21:09:32.178844 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.178849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.179210 (+ 361us) server_negotiation.cc:407] Connection header received
0810 21:09:32.179266 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.179268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.179285 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.179555 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:32.179757 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.180991 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.182212 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.183019 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.183086 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.183143 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.183292 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.183330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":81,"threads_started":1}
I20260810 21:09:32.183744 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.178421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:44017 (local address 127.0.0.1:47294)
0810 21:09:32.178898 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:32.178907 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:32.179076 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:32.179813 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:32.179815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:32.179818 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:32.179859 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.179861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.181049 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.181050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.182179 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.182181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.183118 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.183119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.183189 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.183192 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:32.183226 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 21:09:32.183297 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":4,"thread_start_us":338,"threads_started":1}
I20260810 21:09:32.184078 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.178421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:34953 (local address 127.0.0.1:40180)
0810 21:09:32.178573 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:32.178612 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:32.178813 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:32.180351 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:32.180352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:32.180355 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:32.180391 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.180392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.181830 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.181831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.182626 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.182628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.183375 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.183376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.183436 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.183522 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 21:09:32.183572 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 21:09:32.183617 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":44,"threads_started":1}
I20260810 21:09:32.184237 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.178621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.27.117.66:34953)
0810 21:09:32.180042 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.180046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.180056 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:32.180114 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.180116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.180132 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.180329 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:32.180479 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.181635 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.182707 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.183232 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.183292 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.183342 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.183729 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.183773 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":147,"threads_started":1}
I20260810 21:09:32.184613 29928 tablet_service.cc:1467] Processing CreateTablet for tablet 46892f86751c4711a06a85a75dce170c (DEFAULT_TABLE table=test-table [id=36bcf437544a42598f9d1379c8bfa530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:32.184937 29928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46892f86751c4711a06a85a75dce170c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:32.184845 29693 tablet_service.cc:1467] Processing CreateTablet for tablet 46892f86751c4711a06a85a75dce170c (DEFAULT_TABLE table=test-table [id=36bcf437544a42598f9d1379c8bfa530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:32.184923 29444 tablet_service.cc:1467] Processing CreateTablet for tablet 46892f86751c4711a06a85a75dce170c (DEFAULT_TABLE table=test-table [id=36bcf437544a42598f9d1379c8bfa530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:32.185225 29693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46892f86751c4711a06a85a75dce170c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:32.185225 29444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46892f86751c4711a06a85a75dce170c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:32.188431 30097 tablet_bootstrap.cc:492] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b: Bootstrap starting.
I20260810 21:09:32.189919 30097 tablet_bootstrap.cc:654] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:32.190374 30098 tablet_bootstrap.cc:492] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d: Bootstrap starting.
I20260810 21:09:32.190748 30099 tablet_bootstrap.cc:492] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: Bootstrap starting.
I20260810 21:09:32.191437 30097 log.cc:824] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:32.192005 30098 tablet_bootstrap.cc:654] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:32.192003 30099 tablet_bootstrap.cc:654] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:32.192696 30097 tablet_bootstrap.cc:492] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b: No bootstrap required, opened a new log
I20260810 21:09:32.192873 30097 ts_tablet_manager.cc:1397] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 21:09:32.192965 30099 log.cc:824] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:32.193398 30098 log.cc:824] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:32.193706 30099 tablet_bootstrap.cc:492] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: No bootstrap required, opened a new log
I20260810 21:09:32.193784 30099 ts_tablet_manager.cc:1397] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:09:32.195875 30097 raft_consensus.cc:348] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.196056 30097 raft_consensus.cc:374] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:32.196084 30097 raft_consensus.cc:729] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9023a3569446c89076f5a0f6751a5b, State: Initialized, Role: FOLLOWER
I20260810 21:09:32.196213 30097 consensus_queue.cc:260] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.196651 30097 ts_tablet_manager.cc:1428] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 21:09:32.196485 30099 raft_consensus.cc:348] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.196667 30062 heartbeater.cc:503] Master 127.27.117.126:44765 was elected leader, sending a full tablet report...
I20260810 21:09:32.196729 30099 raft_consensus.cc:374] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:32.196801 30099 raft_consensus.cc:729] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53099f562d214f5ba5b57d626a70039c, State: Initialized, Role: FOLLOWER
I20260810 21:09:32.196950 30099 consensus_queue.cc:260] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.197324 30099 ts_tablet_manager.cc:1428] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260810 21:09:32.197443 29813 heartbeater.cc:503] Master 127.27.117.126:44765 was elected leader, sending a full tablet report...
I20260810 21:09:32.197463 30098 tablet_bootstrap.cc:492] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d: No bootstrap required, opened a new log
I20260810 21:09:32.197547 30098 ts_tablet_manager.cc:1397] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260810 21:09:32.200728 30098 raft_consensus.cc:348] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.201033 30098 raft_consensus.cc:374] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:32.201089 30098 raft_consensus.cc:729] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bbae9767df4b78bf1b825e5192e28d, State: Initialized, Role: FOLLOWER
I20260810 21:09:32.201197 30098 consensus_queue.cc:260] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.201567 30098 ts_tablet_manager.cc:1428] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260810 21:09:32.201575 29568 heartbeater.cc:503] Master 127.27.117.126:44765 was elected leader, sending a full tablet report...
I20260810 21:09:32.208504 30105 raft_consensus.cc:482] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:32.208593 30105 raft_consensus.cc:504] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.209151 30105 leader_election.cc:288] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74bbae9767df4b78bf1b825e5192e28d (127.27.117.65:44017), 53099f562d214f5ba5b57d626a70039c (127.27.117.66:34953)
I20260810 21:09:32.213615 30090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.209430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:47349 (local address 127.27.117.66:34953)
0810 21:09:32.209467 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.209469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.209650 (+ 181us) server_negotiation.cc:407] Connection header received
0810 21:09:32.209760 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.209761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.209769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.209813 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:32.210058 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.210732 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.212531 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.213222 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.213321 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.213328 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.213529 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.213562 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:09:32.213608 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.209423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45003 (local address 127.27.117.65:44017)
0810 21:09:32.209466 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.209468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.209477 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:32.209505 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.209506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.209513 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.209603 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:32.209936 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.210696 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.212325 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.213135 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.213211 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.213220 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.213485 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.213548 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:09:32.213641 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.209282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:44017 (local address 127.27.117.67:45003)
0810 21:09:32.209331 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:32.209341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:32.209382 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:32.209623 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:32.209624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:32.209626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:32.209830 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.209832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.210783 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.210784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.212238 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.212241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.213208 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.213209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.213381 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.213385 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:32.213480 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 21:09:32.213526 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 21:09:32.213881 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.209289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:34953 (local address 127.27.117.67:47349)
0810 21:09:32.209570 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:32.209598 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:32.209674 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:32.209823 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:32.209823 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:32.209825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:32.209996 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.209998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.210800 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.210800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.212421 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:32.212422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.213335 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.213336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:32.213459 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.213464 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:09:32.213515 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:09:32.213548 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":11,"thread_start_us":135,"threads_started":1}
I20260810 21:09:32.214013 29491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46892f86751c4711a06a85a75dce170c" candidate_uuid: "4a9023a3569446c89076f5a0f6751a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bbae9767df4b78bf1b825e5192e28d" is_pre_election: true
I20260810 21:09:32.214238 29491 raft_consensus.cc:2454] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a9023a3569446c89076f5a0f6751a5b in term 0.
I20260810 21:09:32.214334 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46892f86751c4711a06a85a75dce170c" candidate_uuid: "4a9023a3569446c89076f5a0f6751a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53099f562d214f5ba5b57d626a70039c" is_pre_election: true
I20260810 21:09:32.214469 29731 raft_consensus.cc:2454] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a9023a3569446c89076f5a0f6751a5b in term 0.
I20260810 21:09:32.214572 29848 leader_election.cc:302] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9023a3569446c89076f5a0f6751a5b, 74bbae9767df4b78bf1b825e5192e28d; no voters:
I20260810 21:09:32.214689 30105 raft_consensus.cc:2781] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:32.214778 30105 raft_consensus.cc:482] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:32.214808 30105 raft_consensus.cc:3037] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:32.215770 30105 raft_consensus.cc:504] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.215914 30105 leader_election.cc:288] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [CANDIDATE]: Term 1 election: Requested vote from peers 74bbae9767df4b78bf1b825e5192e28d (127.27.117.65:44017), 53099f562d214f5ba5b57d626a70039c (127.27.117.66:34953)
I20260810 21:09:32.216075 29491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46892f86751c4711a06a85a75dce170c" candidate_uuid: "4a9023a3569446c89076f5a0f6751a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74bbae9767df4b78bf1b825e5192e28d"
I20260810 21:09:32.216197 29491 raft_consensus.cc:3037] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:32.216199 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46892f86751c4711a06a85a75dce170c" candidate_uuid: "4a9023a3569446c89076f5a0f6751a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53099f562d214f5ba5b57d626a70039c"
I20260810 21:09:32.216261 29731 raft_consensus.cc:3037] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:32.217819 29731 raft_consensus.cc:2454] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a9023a3569446c89076f5a0f6751a5b in term 1.
I20260810 21:09:32.217819 29491 raft_consensus.cc:2454] T 46892f86751c4711a06a85a75dce170c P 74bbae9767df4b78bf1b825e5192e28d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a9023a3569446c89076f5a0f6751a5b in term 1.
I20260810 21:09:32.218089 29847 leader_election.cc:302] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9023a3569446c89076f5a0f6751a5b, 53099f562d214f5ba5b57d626a70039c; no voters:
I20260810 21:09:32.218209 30105 raft_consensus.cc:2781] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:32.218408 30105 raft_consensus.cc:686] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [term 1 LEADER]: Becoming Leader. State: Replica: 4a9023a3569446c89076f5a0f6751a5b, State: Running, Role: LEADER
I20260810 21:09:32.218518 30105 consensus_queue.cc:237] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } }
I20260810 21:09:32.219588 29214 catalog_manager.cc:5495] T 46892f86751c4711a06a85a75dce170c P 4a9023a3569446c89076f5a0f6751a5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a9023a3569446c89076f5a0f6751a5b (127.27.117.67). New cstate: current_term: 1 leader_uuid: "4a9023a3569446c89076f5a0f6751a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bbae9767df4b78bf1b825e5192e28d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53099f562d214f5ba5b57d626a70039c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a9023a3569446c89076f5a0f6751a5b" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 37635 } health_report { overall_health: HEALTHY } } }
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:32.235882 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.228062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.27.117.65:44017)
0810 21:09:32.228157 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.228160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.228290 (+ 130us) server_negotiation.cc:407] Connection header received
0810 21:09:32.228363 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.228364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.228373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.228418 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:32.229151 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.229610 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.230469 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.230664 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.230741 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.233250 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:32.233258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:32.233259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:32.233265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:32.234904 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.235216 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.235218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.235219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.235273 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.235438 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.235439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.235439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.235563 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:32.235673 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.235726 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.235784 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 21:09:32.236451 29817 tablet.cc:2369] T 46892f86751c4711a06a85a75dce170c P 53099f562d214f5ba5b57d626a70039c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:32.270565 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.262754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.27.117.65:44017)
0810 21:09:32.262802 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.262804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.263567 (+ 763us) server_negotiation.cc:407] Connection header received
0810 21:09:32.263640 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.263641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.263691 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.263748 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:32.264348 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.264984 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.265722 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.265912 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.265984 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.268147 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:32.268148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:32.268148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:32.268155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:32.269678 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.269982 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.269982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.269983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.270034 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.270181 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.270181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.270181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.270257 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:32.270404 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.270438 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.270499 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:32.295425 30086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:32.288608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.27.117.65:44017)
0810 21:09:32.288698 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:32.288700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:32.288850 (+ 150us) server_negotiation.cc:407] Connection header received
0810 21:09:32.288918 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:32.288919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:32.288930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:32.289009 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:32.289548 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.290026 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.290810 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:32.291004 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:32.291106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:32.293386 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:32.293387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:32.293387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:32.293394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:32.294536 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.294842 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.294843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.294843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.294891 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:32.295050 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:32.295050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:32.295051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:32.295132 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:32.295199 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:32.295307 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 21:09:32.295365 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 21:09:32.296187 29417 tablet_service.cc:2908] Rejecting scan request for tablet 46892f86751c4711a06a85a75dce170c: Uninitialized: clean time has not yet been initialized
I20260810 21:09:32.297161 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29331
I20260810 21:09:32.316270 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29573
I20260810 21:09:32.323743 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29822
I20260810 21:09:32.335263 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29158
2026-08-10T21:09:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2238 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[30140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[30140](info): set up 2 sockets
Aug 10 21:09:32 dist-test-slave-2kcd krb5kdc[30140](info): commencing operation
krb5kdc: starting...
W20260810 21:09:33.629660 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:33 dist-test-slave-2kcd krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:33.749284 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41691
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:33.831076 30167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:33.831313 30167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:33.831362 30167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:33.832525 30167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:33.832592 30167 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 21:09:33.832615 30167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:33.832638 30167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:33.832669 30167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:33.832693 30167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:33.834291 30167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41691
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:33.834571 30167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:33.834798 30167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:33.839740 30180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:33.839742 30175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:33.839728 30176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:33.839986 30167 server_base.cc:1034] running on GCE node
I20260810 21:09:33.840282 30167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:33.840611 30167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:33.841785 30167 hybrid_clock.cc:648] HybridClock initialized: now 1786396173841777 us; error 41 us; skew 500 ppm
Aug 10 21:09:33 dist-test-slave-2kcd krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:33.844405 30167 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:33.845060 30167 webserver.cc:460] Webserver started at http://127.27.117.126:36447/ using document root <none> and password file <none>
I20260810 21:09:33.845327 30167 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:33.845404 30167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:33.845566 30167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:33.846686 30167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a5073314ec5649c38006dc7f09291ad3"
format_stamp: "Formatted at 2026-08-10 21:09:33 on dist-test-slave-2kcd"
server_key: "15c4565948ce9d1e08a0268adc6079b1"
server_key_iv: "266cede19d4e1c00566dd7a9dfbc92d2"
server_key_version: "encryptionkey@0"
I20260810 21:09:33.847085 30167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a5073314ec5649c38006dc7f09291ad3"
format_stamp: "Formatted at 2026-08-10 21:09:33 on dist-test-slave-2kcd"
server_key: "15c4565948ce9d1e08a0268adc6079b1"
server_key_iv: "266cede19d4e1c00566dd7a9dfbc92d2"
server_key_version: "encryptionkey@0"
I20260810 21:09:33.848620 30167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:33.849572 30195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:33.849753 30167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:33.849900 30167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a5073314ec5649c38006dc7f09291ad3"
format_stamp: "Formatted at 2026-08-10 21:09:33 on dist-test-slave-2kcd"
server_key: "15c4565948ce9d1e08a0268adc6079b1"
server_key_iv: "266cede19d4e1c00566dd7a9dfbc92d2"
server_key_version: "encryptionkey@0"
I20260810 21:09:33.849983 30167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:33.871007 30167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:33.873034 30167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:33.873216 30167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:33.878834 30167 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:41691
I20260810 21:09:33.878886 30298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:41691 every 8 connection(s)
I20260810 21:09:33.879244 30167 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:33.879901 30300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:33.882323 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: Bootstrap starting.
I20260810 21:09:33.882949 30300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:33.883263 30300 log.cc:824] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:33.884016 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: No bootstrap required, opened a new log
I20260810 21:09:33.884578 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30167
I20260810 21:09:33.884665 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:33.884876 28117 external_mini_cluster.cc:1428] Setting key 3fee7c7362e4b734228a0ca0f64a539b
I20260810 21:09:33.886023 30300 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5073314ec5649c38006dc7f09291ad3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41691 } }
I20260810 21:09:33.886188 30300 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:33.886236 30300 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5073314ec5649c38006dc7f09291ad3, State: Initialized, Role: FOLLOWER
I20260810 21:09:33.886408 30300 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5073314ec5649c38006dc7f09291ad3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41691 } }
I20260810 21:09:33.886512 30300 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:33.886559 30300 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:33.886613 30300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:33.887140 30300 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5073314ec5649c38006dc7f09291ad3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41691 } }
I20260810 21:09:33.887265 30300 leader_election.cc:302] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5073314ec5649c38006dc7f09291ad3; no voters:
I20260810 21:09:33.887456 30300 leader_election.cc:288] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:33.887503 30305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:33.887614 30305 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [term 1 LEADER]: Becoming Leader. State: Replica: a5073314ec5649c38006dc7f09291ad3, State: Running, Role: LEADER
I20260810 21:09:33.887743 30300 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:33.887764 30305 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5073314ec5649c38006dc7f09291ad3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41691 } }
I20260810 21:09:33.888160 30305 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5073314ec5649c38006dc7f09291ad3. Latest consensus state: current_term: 1 leader_uuid: "a5073314ec5649c38006dc7f09291ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5073314ec5649c38006dc7f09291ad3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41691 } } }
I20260810 21:09:33.888168 30306 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5073314ec5649c38006dc7f09291ad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5073314ec5649c38006dc7f09291ad3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41691 } } }
I20260810 21:09:33.888367 30306 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:33.888612 30305 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:33.888952 30317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:33.889413 30317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:33.891171 30317 catalog_manager.cc:1347] Generated new cluster ID: 2cfea38e309748c59993e1539dc5fd87
I20260810 21:09:33.891273 30317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:33 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:33.896960 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:33.885436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.27.117.126:41691)
0810 21:09:33.885651 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:33.885659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:33.885684 (+ 25us) server_negotiation.cc:407] Connection header received
0810 21:09:33.886159 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:33.886165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:33.886210 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:33.886379 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:33.887094 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:33.888454 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:33.889597 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:33.890185 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:33.890324 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:33.893259 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:33.893264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:33.893264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:33.893276 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:33.895183 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:33.895619 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:33.895621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:33.895622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:33.895747 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:33.896009 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:33.896010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:33.896010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:33.896293 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:33.896449 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:33.896579 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 21:09:33.896707 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260810 21:09:33.910876 30317 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:33.911456 30317 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:33.920009 30317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: Generated new TSK 0
I20260810 21:09:33.920202 30317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:33.952157 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41691
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:34.036300 30336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:34.036500 30336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:34.036525 30336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:34.037750 30336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:34.037792 30336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:34.037813 30336 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:09:34.037830 30336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:34.039247 30336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:34.039438 30336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:34.039618 30336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:34.044673 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:34.044680 30350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:34.044680 30354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:34.045174 30336 server_base.cc:1034] running on GCE node
I20260810 21:09:34.045394 30336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:34.045605 30336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:34.046757 30336 hybrid_clock.cc:648] HybridClock initialized: now 1786396174046745 us; error 31 us; skew 500 ppm
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:34.049338 30336 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:34.049801 30336 webserver.cc:460] Webserver started at http://127.27.117.65:36765/ using document root <none> and password file <none>
I20260810 21:09:34.050055 30336 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:34.050100 30336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:34.050264 30336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:34.051311 30336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "8e975bfcc6234932b5bdbe1986ab2dbf"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "07c4dfa106a88952585e0837c3989840"
server_key_iv: "0613fea822555ce9d6bad799fd74f94e"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.051666 30336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "8e975bfcc6234932b5bdbe1986ab2dbf"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "07c4dfa106a88952585e0837c3989840"
server_key_iv: "0613fea822555ce9d6bad799fd74f94e"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.053054 30336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:09:34.053908 30362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:34.054117 30336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:34.054216 30336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "8e975bfcc6234932b5bdbe1986ab2dbf"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "07c4dfa106a88952585e0837c3989840"
server_key_iv: "0613fea822555ce9d6bad799fd74f94e"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.054292 30336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:34.066191 30336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:34.067932 30336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:34.068090 30336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:34.068646 30336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:34.068686 30336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:34.068708 30336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:34.068727 30336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:34.076403 30336 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:35511
I20260810 21:09:34.076491 30571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:35511 every 8 connection(s)
I20260810 21:09:34.076814 30336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:34.077957 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30336
I20260810 21:09:34.078078 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:34.078251 28117 external_mini_cluster.cc:1428] Setting key 2deef58b2c82a3787274221de9b2b26a
I20260810 21:09:34.079079 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.069232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:36959 (local address 127.27.117.126:41691)
0810 21:09:34.069309 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.069313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.069506 (+ 193us) server_negotiation.cc:407] Connection header received
0810 21:09:34.069974 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.069975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.069993 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.070076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.070626 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.071475 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.072524 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.072910 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.073000 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.075709 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.075710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.075711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.075721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.077575 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.077996 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.077997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.077998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.078062 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.078327 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.078328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.078328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.078485 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.078563 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.078893 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.079002 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 21:09:34.079277 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.069155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.27.117.65:36959)
0810 21:09:34.069361 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.069402 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.069876 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.070145 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.070148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.070373 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.070551 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.070555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.071592 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.071593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.072456 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.072458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.073031 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.073032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.073166 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.073656 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.073664 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.075607 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.077662 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.077669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.077672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.077918 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.078133 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.078134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.078135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.078251 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.078604 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.078610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.078723 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.078853 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.079021 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260810 21:09:34.079762 30572 heartbeater.cc:347] Connected to a master server at 127.27.117.126:41691
I20260810 21:09:34.079861 30572 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:34.080049 30572 heartbeater.cc:511] Master 127.27.117.126:41691 requested a full tablet report, sending...
I20260810 21:09:34.081067 30217 ts_manager.cc:194] Registered new tserver with Master: 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511)
I20260810 21:09:34.082006 30217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:36959
I20260810 21:09:34.087495 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.080965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.27.117.65:46425)
0810 21:09:34.081895 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.081909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.081975 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.082092 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.082092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.082302 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.082369 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.082371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.083025 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.083026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.083539 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.083541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.083925 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.083926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.084004 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.084437 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.084441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.084702 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.086453 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.086454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.086454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.086614 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.086832 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.086832 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.086833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.086909 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.087187 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.087188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.087264 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.087320 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.087432 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":75,"threads_started":1}
I20260810 21:09:34.087560 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.081012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:46425 (local address 127.27.117.126:41691)
0810 21:09:34.081049 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.081051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.081983 (+ 932us) server_negotiation.cc:407] Connection header received
0810 21:09:34.082016 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.082017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.082028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.082087 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.082450 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.082925 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.083629 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.083839 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.083915 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.084812 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.084813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.084814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.084822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.086368 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.086692 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.086693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.086693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.086759 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.086977 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.086978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.086979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.087115 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.087194 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.087370 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.087454 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:34.107118 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41691
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:34.188259 30587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:34.188489 30587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:34.188532 30587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:34.189750 30587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:34.189821 30587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:34.189857 30587 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:09:34.189898 30587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:34.191313 30587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:34.191577 30587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:34.191813 30587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:34.196630 30599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:34.196647 30596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:34.196699 30597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:34.196797 30587 server_base.cc:1034] running on GCE node
I20260810 21:09:34.197139 30587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:34.197393 30587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:34.198534 30587 hybrid_clock.cc:648] HybridClock initialized: now 1786396174198514 us; error 36 us; skew 500 ppm
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:34.201117 30587 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:34.201612 30587 webserver.cc:460] Webserver started at http://127.27.117.66:44711/ using document root <none> and password file <none>
I20260810 21:09:34.201839 30587 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:34.201893 30587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:34.202028 30587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:34.203047 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "bfc459a3720c4658a308c6762854fd4e"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "d934587fff575631c22eb571b36b53a8"
server_key_iv: "b9d049d0ea114ee42febc1ac44e47d6f"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.203408 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "bfc459a3720c4658a308c6762854fd4e"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "d934587fff575631c22eb571b36b53a8"
server_key_iv: "b9d049d0ea114ee42febc1ac44e47d6f"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.204870 30587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:34.205624 30621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:34.205816 30587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:34.205938 30587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "bfc459a3720c4658a308c6762854fd4e"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "d934587fff575631c22eb571b36b53a8"
server_key_iv: "b9d049d0ea114ee42febc1ac44e47d6f"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.206017 30587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:34.234999 30587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:34.237332 30587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:34.237499 30587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:34.238015 30587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:34.238102 30587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:34.238143 30587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:34.238173 30587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:34.245538 30587 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:43921
I20260810 21:09:34.245609 30830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:43921 every 8 connection(s)
I20260810 21:09:34.245927 30587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:34.248446 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.238513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:43119 (local address 127.27.117.126:41691)
0810 21:09:34.238588 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.238590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.238786 (+ 196us) server_negotiation.cc:407] Connection header received
0810 21:09:34.239231 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.239233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.239247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.239317 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.240192 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.241085 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.242382 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.242701 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.242773 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.245506 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.245508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.245508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.245516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.247053 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.247463 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.247464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.247464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.247524 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.247791 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.247792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.247793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.247917 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.247988 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.248274 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.248360 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 21:09:34.248620 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.238470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.27.117.66:43119)
0810 21:09:34.238648 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.238687 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.239136 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.239392 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.239395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.239772 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.240062 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.240069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.241191 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.241192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.242284 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.242287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.242819 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.242821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.242993 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.243536 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.243545 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.245408 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.247152 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.247159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.247161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.247381 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.247603 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.247604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.247604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.247719 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.247992 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.247999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.248135 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.248235 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.248385 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
I20260810 21:09:34.249146 30837 heartbeater.cc:347] Connected to a master server at 127.27.117.126:41691
I20260810 21:09:34.249234 30837 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:34.249433 30837 heartbeater.cc:511] Master 127.27.117.126:41691 requested a full tablet report, sending...
I20260810 21:09:34.249827 30217 ts_manager.cc:194] Registered new tserver with Master: bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.250340 30217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:43119
I20260810 21:09:34.252205 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30587
I20260810 21:09:34.252300 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:34.252555 28117 external_mini_cluster.cc:1428] Setting key f31e7255d57d7c1be8049f5b99417982
I20260810 21:09:34.257658 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.250245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.27.117.66:42997)
0810 21:09:34.250395 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.250434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.250520 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.250661 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.250662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.250902 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.250974 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.250976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.251998 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.251998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.252969 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.252972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.253526 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.253527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.253616 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.254040 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.254045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.254393 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.256293 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.256294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.256294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.256580 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.256867 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.256867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.256868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.256935 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.257318 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.257319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.257379 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.257452 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.257562 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260810 21:09:34.257741 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.250479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:42997 (local address 127.27.117.126:41691)
0810 21:09:34.250514 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.250515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.250525 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:34.250585 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.250586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.250596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.250651 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.251080 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.251870 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.253088 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.253406 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.253449 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.254484 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.254485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.254485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.254492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.256197 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.256662 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.256663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.256663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.256759 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.257010 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.257011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.257012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.257235 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.257303 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.257517 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.257619 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:34.279889 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41691
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:34.366863 30853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:34.367070 30853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:34.367094 30853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:34.368211 30853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:34.368248 30853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:34.368268 30853 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:09:34.368285 30853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:34.369764 30853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:34.369968 30853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:34.370141 30853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:34.375207 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:34.375207 30862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:34.375265 30870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:34.375597 30853 server_base.cc:1034] running on GCE node
I20260810 21:09:34.375804 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:34.376078 30853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:34.377238 30853 hybrid_clock.cc:648] HybridClock initialized: now 1786396174377226 us; error 45 us; skew 500 ppm
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:34.385636 30853 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:34.386157 30853 webserver.cc:460] Webserver started at http://127.27.117.67:41793/ using document root <none> and password file <none>
I20260810 21:09:34.386405 30853 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:34.386477 30853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:34.386719 30853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:34.387849 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "522a624ecf05418ca34fc8be9e39621f"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "306dc47ca4e190218d55e72829f6043a"
server_key_iv: "2b4dfdfb3f5b2124ea3db3fc0a59b716"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.388235 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "522a624ecf05418ca34fc8be9e39621f"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "306dc47ca4e190218d55e72829f6043a"
server_key_iv: "2b4dfdfb3f5b2124ea3db3fc0a59b716"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.389838 30853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:34.390609 30883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:34.390810 30853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:34.390893 30853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "522a624ecf05418ca34fc8be9e39621f"
format_stamp: "Formatted at 2026-08-10 21:09:34 on dist-test-slave-2kcd"
server_key: "306dc47ca4e190218d55e72829f6043a"
server_key_iv: "2b4dfdfb3f5b2124ea3db3fc0a59b716"
server_key_version: "encryptionkey@0"
I20260810 21:09:34.390982 30853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:34.409578 30853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:34.411487 30853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:34.411639 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:34.412173 30853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:34.412209 30853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:34.412237 30853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:34.412289 30853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:34.419472 30853 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:33531
I20260810 21:09:34.419559 31096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:33531 every 8 connection(s)
I20260810 21:09:34.419900 30853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:34.422421 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.412744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:39711 (local address 127.27.117.126:41691)
0810 21:09:34.412853 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.412856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.413042 (+ 186us) server_negotiation.cc:407] Connection header received
0810 21:09:34.413534 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.413535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.413550 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.413627 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.414515 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.415356 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.416984 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.417202 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.417282 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.419613 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.419615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.419615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.419623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.421276 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.421647 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.421647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.421648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.421699 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.421911 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.421912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.421912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.422012 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.422068 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.422291 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.422355 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 21:09:34.422575 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.412698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.27.117.67:39711)
0810 21:09:34.412906 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.412948 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.413433 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.413723 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.413728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.414120 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.414401 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.414407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.415495 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.415496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.416866 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.416869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.417309 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.417310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.417442 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.417968 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.417977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.419522 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.421347 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.421353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.421355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.421576 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.421759 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.421760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.421761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.421856 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.422076 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.422080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.422189 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.422270 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.422396 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":123,"threads_started":1}
I20260810 21:09:34.423045 31097 heartbeater.cc:347] Connected to a master server at 127.27.117.126:41691
I20260810 21:09:34.423127 31097 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:34.423316 31097 heartbeater.cc:511] Master 127.27.117.126:41691 requested a full tablet report, sending...
I20260810 21:09:34.424023 30218 ts_manager.cc:194] Registered new tserver with Master: 522a624ecf05418ca34fc8be9e39621f (127.27.117.67:33531)
I20260810 21:09:34.424422 30218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:39711
I20260810 21:09:34.424860 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30853
I20260810 21:09:34.424966 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:34.425163 28117 external_mini_cluster.cc:1428] Setting key 1a47ee568ecbba0ba77fcd0203dc2e10
I20260810 21:09:34.425998 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:09:34.430256 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.424135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:47323 (local address 127.27.117.126:41691)
0810 21:09:34.424172 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.424174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.424677 (+ 503us) server_negotiation.cc:407] Connection header received
0810 21:09:34.424704 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.424705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.424716 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.424773 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.425140 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.425719 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.426425 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.426735 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.426803 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.427625 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.427626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.427627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.427634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.429195 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.429601 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.429602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.429602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.429650 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.429850 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.429851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.429851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.429941 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.429990 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.430150 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.430201 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 21:09:34.430387 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.424057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.27.117.67:47323)
0810 21:09:34.424584 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.424596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.424661 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.424806 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.424807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.425003 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.425058 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.425060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.425800 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.425801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.426346 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.426348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.426817 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.426818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.426869 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.427284 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.427287 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.427541 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.429279 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.429280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.429280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.429527 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.429717 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.429718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.429719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.429790 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.430005 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.430005 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.430054 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.430135 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.430247 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":65,"threads_started":1}
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:34.450721 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.444107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.27.117.126:41691)
0810 21:09:34.444171 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.444173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.444181 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:34.444204 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.444204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.444212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.444266 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.444854 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.445317 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.446065 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.446237 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.446285 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.448367 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.448368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.448368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.448374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.449745 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.450059 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.450059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.450060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.450107 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.450330 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.450330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.450331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.450435 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.450512 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.450606 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.450659 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:09:34.452253 30218 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:34.453974 30218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:34.460592 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.456543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43921 (local address 127.0.0.1:52504)
0810 21:09:34.456848 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.456857 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.456894 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.457455 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.457456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.457458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:34.457490 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.457492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.458726 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.458727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.459608 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.459611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.460303 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.460304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.460378 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.460391 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.460456 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.460503 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
I20260810 21:09:34.460768 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.456353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.27.117.67:33531)
0810 21:09:34.456663 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.456666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.456804 (+ 138us) server_negotiation.cc:407] Connection header received
0810 21:09:34.457009 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.457012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.457029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.457119 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:34.457372 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.458243 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.459671 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.460386 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.460464 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.460519 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.460580 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.460629 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260810 21:09:34.460764 31137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.456236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:33531 (local address 127.0.0.1:59338)
0810 21:09:34.456712 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.456722 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.456851 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.457162 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.457165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.457169 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:34.457278 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.457279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.458324 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.458324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.459577 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.459580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.460462 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.460463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.460513 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.460516 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.460567 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.460599 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260810 21:09:34.461014 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.456236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:35511 (local address 127.0.0.1:38154)
0810 21:09:34.456589 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.456632 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.456719 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.457162 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.457165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.457167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:34.457273 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.457275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.458647 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.458648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.459708 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.459710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.460574 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.460575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.460654 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.460658 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.460759 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.460805 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":59,"thread_start_us":60,"threads_started":1}
I20260810 21:09:34.461047 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.456379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.27.117.65:35511)
0810 21:09:34.456572 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.456578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.456701 (+ 123us) server_negotiation.cc:407] Connection header received
0810 21:09:34.456956 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.456959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.456977 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.457075 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:34.457354 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.458561 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.459778 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.460425 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.460482 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.460526 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.460874 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.460924 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260810 21:09:34.461452 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.456667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.27.117.66:43921)
0810 21:09:34.457285 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.457289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.457298 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:34.457354 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.457356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.457371 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.457460 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:34.457579 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.458655 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.459703 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.460213 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.460304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.460365 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.461291 (+ 926us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.461347 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":90,"threads_started":1}
I20260810 21:09:34.461771 30451 tablet_service.cc:1467] Processing CreateTablet for tablet 85a651a6f2ff44988f3451bcd9eb060c (DEFAULT_TABLE table=test-table [id=30d637f53fef46088f75b7f1e10dec2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:34.461774 30978 tablet_service.cc:1467] Processing CreateTablet for tablet 85a651a6f2ff44988f3451bcd9eb060c (DEFAULT_TABLE table=test-table [id=30d637f53fef46088f75b7f1e10dec2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:34.462019 30710 tablet_service.cc:1467] Processing CreateTablet for tablet 85a651a6f2ff44988f3451bcd9eb060c (DEFAULT_TABLE table=test-table [id=30d637f53fef46088f75b7f1e10dec2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:34.462141 30978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a651a6f2ff44988f3451bcd9eb060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.462141 30451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a651a6f2ff44988f3451bcd9eb060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.462271 30710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a651a6f2ff44988f3451bcd9eb060c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.465376 31141 tablet_bootstrap.cc:492] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Bootstrap starting.
I20260810 21:09:34.465695 31145 tablet_bootstrap.cc:492] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Bootstrap starting.
I20260810 21:09:34.465744 31143 tablet_bootstrap.cc:492] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Bootstrap starting.
I20260810 21:09:34.466169 31141 tablet_bootstrap.cc:654] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.466325 31145 tablet_bootstrap.cc:654] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.466573 31141 log.cc:824] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:34.466660 31145 log.cc:824] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:34.466857 31143 tablet_bootstrap.cc:654] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.467283 31143 log.cc:824] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:34.467443 31145 tablet_bootstrap.cc:492] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: No bootstrap required, opened a new log
I20260810 21:09:34.467443 31141 tablet_bootstrap.cc:492] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: No bootstrap required, opened a new log
I20260810 21:09:34.467514 31141 ts_tablet_manager.cc:1397] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:34.467514 31145 ts_tablet_manager.cc:1397] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:34.467952 31143 tablet_bootstrap.cc:492] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: No bootstrap required, opened a new log
I20260810 21:09:34.468035 31143 ts_tablet_manager.cc:1397] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:34.469506 31145 raft_consensus.cc:348] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.469506 31141 raft_consensus.cc:348] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.469571 31143 raft_consensus.cc:348] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.469683 31145 raft_consensus.cc:374] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.469683 31141 raft_consensus.cc:374] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.469717 31143 raft_consensus.cc:374] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.469723 31141 raft_consensus.cc:729] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc459a3720c4658a308c6762854fd4e, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.469723 31145 raft_consensus.cc:729] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e975bfcc6234932b5bdbe1986ab2dbf, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.469766 31143 raft_consensus.cc:729] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522a624ecf05418ca34fc8be9e39621f, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.469849 31141 consensus_queue.cc:260] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.469864 31145 consensus_queue.cc:260] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.469913 31143 consensus_queue.cc:260] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.470146 31141 ts_tablet_manager.cc:1428] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:09:34.470146 31145 ts_tablet_manager.cc:1428] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:09:34.470185 30837 heartbeater.cc:503] Master 127.27.117.126:41691 was elected leader, sending a full tablet report...
I20260810 21:09:34.470199 31143 ts_tablet_manager.cc:1428] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:34.470266 30572 heartbeater.cc:503] Master 127.27.117.126:41691 was elected leader, sending a full tablet report...
I20260810 21:09:34.470304 31097 heartbeater.cc:503] Master 127.27.117.126:41691 was elected leader, sending a full tablet report...
I20260810 21:09:34.471112 31153 raft_consensus.cc:482] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:34.471194 31153 raft_consensus.cc:504] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.471487 31153 leader_election.cc:288] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511), bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.474809 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.471681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:35511 (local address 127.27.117.67:33097)
0810 21:09:34.471721 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.471734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.471781 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.471908 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.471908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.471910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:34.471967 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.471968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.472830 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.472831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.474052 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.474055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.474619 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.474620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.474688 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.474691 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.474733 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.474763 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 21:09:34.474848 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.471730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:33097 (local address 127.27.117.65:35511)
0810 21:09:34.471765 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.471767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.471776 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:34.471842 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.471843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.471851 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.471904 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:34.472026 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.472706 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.474137 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.474544 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.474614 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.474622 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.474755 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.474789 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:09:34.475224 30485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a651a6f2ff44988f3451bcd9eb060c" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" is_pre_election: true
I20260810 21:09:34.475273 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.472003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:39213 (local address 127.27.117.66:43921)
0810 21:09:34.472094 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.472096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.472104 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:34.472217 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.472218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.472228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.472277 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:34.472395 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.472999 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.474368 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.474881 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.474950 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.474958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.475177 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.475214 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 21:09:34.475286 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.471783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43921 (local address 127.27.117.67:39213)
0810 21:09:34.471902 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.471913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.472129 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.472286 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.472286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.472288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:34.472338 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.472339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.473064 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.473065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.474297 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.474298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.475008 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.475009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.475090 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.475093 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.475172 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.475208 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260810 21:09:34.475371 30485 raft_consensus.cc:2454] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 0.
I20260810 21:09:34.475605 30888 leader_election.cc:302] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522a624ecf05418ca34fc8be9e39621f, 8e975bfcc6234932b5bdbe1986ab2dbf; no voters:
I20260810 21:09:34.475756 31153 raft_consensus.cc:2781] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:34.475823 31153 raft_consensus.cc:482] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:34.475849 31153 raft_consensus.cc:3037] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.475920 30744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a651a6f2ff44988f3451bcd9eb060c" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc459a3720c4658a308c6762854fd4e" is_pre_election: true
I20260810 21:09:34.476040 30744 raft_consensus.cc:2454] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 0.
I20260810 21:09:34.476543 31153 raft_consensus.cc:504] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.476677 31153 leader_election.cc:288] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 election: Requested vote from peers 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511), bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.476845 30485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a651a6f2ff44988f3451bcd9eb060c" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf"
I20260810 21:09:34.476858 30744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a651a6f2ff44988f3451bcd9eb060c" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc459a3720c4658a308c6762854fd4e"
I20260810 21:09:34.476945 30744 raft_consensus.cc:3037] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.476923 30485 raft_consensus.cc:3037] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.478015 30485 raft_consensus.cc:2454] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 1.
I20260810 21:09:34.478034 30744 raft_consensus.cc:2454] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 1.
I20260810 21:09:34.478214 30888 leader_election.cc:302] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522a624ecf05418ca34fc8be9e39621f, bfc459a3720c4658a308c6762854fd4e; no voters:
I20260810 21:09:34.478322 31153 raft_consensus.cc:2781] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:34.478475 31153 raft_consensus.cc:686] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 1 LEADER]: Becoming Leader. State: Replica: 522a624ecf05418ca34fc8be9e39621f, State: Running, Role: LEADER
I20260810 21:09:34.478580 31153 consensus_queue.cc:237] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:09:34.479413 30217 catalog_manager.cc:5495] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f reported cstate change: term changed from 0 to 1, leader changed from <none> to 522a624ecf05418ca34fc8be9e39621f (127.27.117.67). New cstate: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } health_report { overall_health: UNKNOWN } } }
I20260810 21:09:34.489315 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.486824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.27.117.67:33531)
0810 21:09:34.486892 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.486894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.486905 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:34.486991 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.486992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.487005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.487068 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:34.487262 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.487732 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.488576 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.488750 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.488864 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.489060 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:34.489140 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:34.489194 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.489239 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.489270 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 21:09:34.491472 30485 raft_consensus.cc:1261] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Refusing update from remote peer 522a624ecf05418ca34fc8be9e39621f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:34.491472 30744 raft_consensus.cc:1261] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Refusing update from remote peer 522a624ecf05418ca34fc8be9e39621f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:34.491864 31153 consensus_queue.cc:1035] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:34.492043 31156 consensus_queue.cc:1035] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:34.495072 31160 mvcc.cc:205] Tried to move back new op lower bound from 7317078730714398720 to 7317078730665070592. Current Snapshot: MvccSnapshot[applied={T|T < 7317078730714398720}]
I20260810 21:09:34.499729 30217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:34.499859 30217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:34.500685 30217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: Sending DeleteTablet for 3 replicas of tablet 85a651a6f2ff44988f3451bcd9eb060c
I20260810 21:09:34.501231 30978 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a651a6f2ff44988f3451bcd9eb060c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:59338
I20260810 21:09:34.501231 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a651a6f2ff44988f3451bcd9eb060c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:52504
I20260810 21:09:34.501631 30451 tablet_service.cc:1514] Processing DeleteTablet for tablet 85a651a6f2ff44988f3451bcd9eb060c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:38154
I20260810 21:09:34.501760 30217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:34.501952 30217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:34.501981 31181 tablet_replica.cc:330] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: stopping tablet replica
I20260810 21:09:34.501885 31179 tablet_replica.cc:330] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: stopping tablet replica
I20260810 21:09:34.502094 31181 raft_consensus.cc:2229] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:34.502233 31179 raft_consensus.cc:2229] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:34.502362 31179 raft_consensus.cc:2258] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:34.502543 31177 tablet_replica.cc:330] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: stopping tablet replica
I20260810 21:09:34.502556 31181 raft_consensus.cc:2258] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:34.502637 31177 raft_consensus.cc:2229] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:34.502803 31177 raft_consensus.cc:2258] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:34.503305 31177 ts_tablet_manager.cc:1905] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:34.503477 30451 tablet_service.cc:1467] Processing CreateTablet for tablet 36e7e662f74140e398480281d964c38b (DEFAULT_TABLE table=test-table [id=3bbbe46d72224c2db39df390f2d9c248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:34.503625 30451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e7e662f74140e398480281d964c38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.505138 31179 ts_tablet_manager.cc:1905] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:34.505901 31145 tablet_bootstrap.cc:492] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Bootstrap starting.
I20260810 21:09:34.506019 31177 ts_tablet_manager.cc:1918] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:34.506067 30710 tablet_service.cc:1467] Processing CreateTablet for tablet 36e7e662f74140e398480281d964c38b (DEFAULT_TABLE table=test-table [id=3bbbe46d72224c2db39df390f2d9c248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:34.506237 30710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e7e662f74140e398480281d964c38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.506280 31177 log.cc:1196] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/85a651a6f2ff44988f3451bcd9eb060c
I20260810 21:09:34.507007 31145 tablet_bootstrap.cc:654] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.507422 31177 ts_tablet_manager.cc:1939] T 85a651a6f2ff44988f3451bcd9eb060c P bfc459a3720c4658a308c6762854fd4e: Deleting consensus metadata
I20260810 21:09:34.507448 31179 ts_tablet_manager.cc:1918] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:34.507513 31179 log.cc:1196] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/85a651a6f2ff44988f3451bcd9eb060c
I20260810 21:09:34.507822 31179 ts_tablet_manager.cc:1939] T 85a651a6f2ff44988f3451bcd9eb060c P 522a624ecf05418ca34fc8be9e39621f: Deleting consensus metadata
I20260810 21:09:34.507969 31141 tablet_bootstrap.cc:492] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Bootstrap starting.
I20260810 21:09:34.508198 31145 tablet_bootstrap.cc:492] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: No bootstrap required, opened a new log
I20260810 21:09:34.508194 30196 catalog_manager.cc:4841] TS 522a624ecf05418ca34fc8be9e39621f (127.27.117.67:33531): tablet 85a651a6f2ff44988f3451bcd9eb060c (table test-table [id=30d637f53fef46088f75b7f1e10dec2f]) successfully deleted
I20260810 21:09:34.508263 31145 ts_tablet_manager.cc:1397] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 21:09:34.508283 30978 tablet_service.cc:1467] Processing CreateTablet for tablet 36e7e662f74140e398480281d964c38b (DEFAULT_TABLE table=test-table [id=3bbbe46d72224c2db39df390f2d9c248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:34.508378 30978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36e7e662f74140e398480281d964c38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.508538 31181 ts_tablet_manager.cc:1905] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:34.508749 31145 raft_consensus.cc:348] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.508859 31145 raft_consensus.cc:374] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.508888 31145 raft_consensus.cc:729] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e975bfcc6234932b5bdbe1986ab2dbf, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.508884 31141 tablet_bootstrap.cc:654] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.508955 31145 consensus_queue.cc:260] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.509117 31145 ts_tablet_manager.cc:1428] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:34.509920 31143 tablet_bootstrap.cc:492] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Bootstrap starting.
I20260810 21:09:34.510430 31141 tablet_bootstrap.cc:492] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: No bootstrap required, opened a new log
I20260810 21:09:34.510524 31141 ts_tablet_manager.cc:1397] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 21:09:34.510941 31141 raft_consensus.cc:348] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.511008 31143 tablet_bootstrap.cc:654] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.511026 31141 raft_consensus.cc:374] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.511070 31141 raft_consensus.cc:729] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc459a3720c4658a308c6762854fd4e, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.511107 31141 consensus_queue.cc:260] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.511198 31141 ts_tablet_manager.cc:1428] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:34.511988 30198 catalog_manager.cc:4841] TS bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921): tablet 85a651a6f2ff44988f3451bcd9eb060c (table test-table [id=30d637f53fef46088f75b7f1e10dec2f]) successfully deleted
I20260810 21:09:34.512300 31143 tablet_bootstrap.cc:492] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: No bootstrap required, opened a new log
I20260810 21:09:34.512349 31143 ts_tablet_manager.cc:1397] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:34.512687 31143 raft_consensus.cc:348] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.512789 31143 raft_consensus.cc:374] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.512835 31143 raft_consensus.cc:729] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522a624ecf05418ca34fc8be9e39621f, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.512878 31143 consensus_queue.cc:260] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.512988 31143 ts_tablet_manager.cc:1428] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:34.513644 31181 ts_tablet_manager.cc:1918] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:34.513751 31181 log.cc:1196] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/85a651a6f2ff44988f3451bcd9eb060c
I20260810 21:09:34.514030 31181 ts_tablet_manager.cc:1939] T 85a651a6f2ff44988f3451bcd9eb060c P 8e975bfcc6234932b5bdbe1986ab2dbf: Deleting consensus metadata
I20260810 21:09:34.514533 30198 catalog_manager.cc:4841] TS 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511): tablet 85a651a6f2ff44988f3451bcd9eb060c (table test-table [id=30d637f53fef46088f75b7f1e10dec2f]) successfully deleted
W20260810 21:09:34.577672 30574 tablet.cc:2369] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:34.619578 31174 raft_consensus.cc:482] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:34.619695 31174 raft_consensus.cc:504] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.619899 31174 leader_election.cc:288] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511), bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.620200 30744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e7e662f74140e398480281d964c38b" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc459a3720c4658a308c6762854fd4e" is_pre_election: true
I20260810 21:09:34.620244 30485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e7e662f74140e398480281d964c38b" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" is_pre_election: true
I20260810 21:09:34.620319 30744 raft_consensus.cc:2454] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 0.
I20260810 21:09:34.620370 30485 raft_consensus.cc:2454] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 0.
I20260810 21:09:34.620488 30888 leader_election.cc:302] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522a624ecf05418ca34fc8be9e39621f, bfc459a3720c4658a308c6762854fd4e; no voters:
I20260810 21:09:34.620640 31174 raft_consensus.cc:2781] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:34.620718 31174 raft_consensus.cc:482] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:34.620739 31174 raft_consensus.cc:3037] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.622102 31174 raft_consensus.cc:504] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.622249 31174 leader_election.cc:288] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 election: Requested vote from peers 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511), bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.622521 30485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e7e662f74140e398480281d964c38b" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf"
I20260810 21:09:34.622514 30744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36e7e662f74140e398480281d964c38b" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc459a3720c4658a308c6762854fd4e"
I20260810 21:09:34.622612 30485 raft_consensus.cc:3037] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.622612 30744 raft_consensus.cc:3037] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.623755 30485 raft_consensus.cc:2454] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 1.
I20260810 21:09:34.623755 30744 raft_consensus.cc:2454] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 1.
I20260810 21:09:34.623956 30888 leader_election.cc:302] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522a624ecf05418ca34fc8be9e39621f, bfc459a3720c4658a308c6762854fd4e; no voters:
I20260810 21:09:34.624070 31174 raft_consensus.cc:2781] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:34.624130 31174 raft_consensus.cc:686] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 1 LEADER]: Becoming Leader. State: Replica: 522a624ecf05418ca34fc8be9e39621f, State: Running, Role: LEADER
I20260810 21:09:34.624188 31174 consensus_queue.cc:237] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.624854 30217 catalog_manager.cc:5495] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f reported cstate change: term changed from 0 to 1, leader changed from <none> to 522a624ecf05418ca34fc8be9e39621f (127.27.117.67). New cstate: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } health_report { overall_health: HEALTHY } } }
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396173, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
W20260810 21:09:34.670562 31101 tablet.cc:2369] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:34.672595 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.665695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.27.117.126:41691)
0810 21:09:34.665778 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.665780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.665977 (+ 197us) server_negotiation.cc:407] Connection header received
0810 21:09:34.666028 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.666029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.666039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.666092 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.666641 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.667116 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.667785 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.667957 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.668005 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.670155 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.670157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.670157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.670163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.671634 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.671971 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.671972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.671972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.672021 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.672197 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.672198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.672198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.672292 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.672342 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.672460 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.672524 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 21:09:34.672753 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.665634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41691 (local address 127.0.0.1:37084)
0810 21:09:34.665899 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.665910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.665975 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.666113 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.666114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.666521 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:34.666570 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.666572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.667188 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.667189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.667714 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.667715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.668021 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.668022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.668063 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.668588 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:34.668592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:34.670076 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:34.671708 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.671715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.671717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.671907 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.672086 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:34.672086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:34.672086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:34.672145 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:34.672349 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:34.672355 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:34.672394 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.672446 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.672595 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":61,"thread_start_us":189,"threads_started":1}
I20260810 21:09:34.674955 30217 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:37084:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 21:09:34.675241 30217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:34.676877 30451 tablet_service.cc:1467] Processing CreateTablet for tablet 83b2a626c6ce40e1b6c14f3b0003e96f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a6ac85163a346148dd0eb63beaca8a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:09:34.676836 30978 tablet_service.cc:1467] Processing CreateTablet for tablet 83b2a626c6ce40e1b6c14f3b0003e96f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a6ac85163a346148dd0eb63beaca8a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:09:34.677069 30451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b2a626c6ce40e1b6c14f3b0003e96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.677070 30978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b2a626c6ce40e1b6c14f3b0003e96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.678701 31143 tablet_bootstrap.cc:492] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: Bootstrap starting.
I20260810 21:09:34.678813 30710 tablet_service.cc:1467] Processing CreateTablet for tablet 83b2a626c6ce40e1b6c14f3b0003e96f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5a6ac85163a346148dd0eb63beaca8a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:09:34.678962 30710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b2a626c6ce40e1b6c14f3b0003e96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:34.679152 31143 tablet_bootstrap.cc:654] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.679490 31145 tablet_bootstrap.cc:492] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: Bootstrap starting.
I20260810 21:09:34.680104 31143 tablet_bootstrap.cc:492] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: No bootstrap required, opened a new log
I20260810 21:09:34.680169 31143 ts_tablet_manager.cc:1397] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:34.680429 31143 raft_consensus.cc:348] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.680524 31143 raft_consensus.cc:374] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.680599 31143 raft_consensus.cc:729] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522a624ecf05418ca34fc8be9e39621f, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.680608 31145 tablet_bootstrap.cc:654] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.680655 31141 tablet_bootstrap.cc:492] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: Bootstrap starting.
I20260810 21:09:34.680648 31143 consensus_queue.cc:260] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.680830 31143 ts_tablet_manager.cc:1428] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:34.681429 31174 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.681489 31141 tablet_bootstrap.cc:654] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:34.681536 31174 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:34.681872 31145 tablet_bootstrap.cc:492] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: No bootstrap required, opened a new log
I20260810 21:09:34.681950 31145 ts_tablet_manager.cc:1397] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 21:09:34.682385 31145 raft_consensus.cc:348] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.682472 31145 raft_consensus.cc:374] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.682497 31145 raft_consensus.cc:729] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e975bfcc6234932b5bdbe1986ab2dbf, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.682543 31141 tablet_bootstrap.cc:492] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: No bootstrap required, opened a new log
I20260810 21:09:34.682559 31145 consensus_queue.cc:260] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.682596 31141 ts_tablet_manager.cc:1397] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 21:09:34.682652 31145 ts_tablet_manager.cc:1428] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:34.682838 31152 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.682940 31152 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:34.683100 31141 raft_consensus.cc:348] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.683266 31141 raft_consensus.cc:374] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:34.683305 31141 raft_consensus.cc:729] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc459a3720c4658a308c6762854fd4e, State: Initialized, Role: FOLLOWER
I20260810 21:09:34.683370 31141 consensus_queue.cc:260] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.683463 31141 ts_tablet_manager.cc:1428] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:34.683581 31150 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.683729 31150 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:34.732064 31174 raft_consensus.cc:482] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:34.732163 31174 raft_consensus.cc:504] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.732352 31174 leader_election.cc:288] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511), bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.732589 30485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b2a626c6ce40e1b6c14f3b0003e96f" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" is_pre_election: true
I20260810 21:09:34.732710 30485 raft_consensus.cc:2454] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 0.
I20260810 21:09:34.732776 30744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b2a626c6ce40e1b6c14f3b0003e96f" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc459a3720c4658a308c6762854fd4e" is_pre_election: true
I20260810 21:09:34.732913 30744 raft_consensus.cc:2454] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 0.
I20260810 21:09:34.732913 30888 leader_election.cc:302] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522a624ecf05418ca34fc8be9e39621f, 8e975bfcc6234932b5bdbe1986ab2dbf; no voters:
I20260810 21:09:34.733007 31174 raft_consensus.cc:2781] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:34.733084 31174 raft_consensus.cc:482] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:34.733124 31174 raft_consensus.cc:3037] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.733839 31174 raft_consensus.cc:504] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.733981 31174 leader_election.cc:288] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 election: Requested vote from peers 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511), bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921)
I20260810 21:09:34.734157 30485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b2a626c6ce40e1b6c14f3b0003e96f" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf"
I20260810 21:09:34.734230 30485 raft_consensus.cc:3037] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.734164 30744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b2a626c6ce40e1b6c14f3b0003e96f" candidate_uuid: "522a624ecf05418ca34fc8be9e39621f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfc459a3720c4658a308c6762854fd4e"
I20260810 21:09:34.734261 30744 raft_consensus.cc:3037] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:34.734936 30485 raft_consensus.cc:2454] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 1.
I20260810 21:09:34.735118 30744 raft_consensus.cc:2454] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 522a624ecf05418ca34fc8be9e39621f in term 1.
I20260810 21:09:34.735147 30888 leader_election.cc:302] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 522a624ecf05418ca34fc8be9e39621f, 8e975bfcc6234932b5bdbe1986ab2dbf; no voters:
I20260810 21:09:34.735281 31174 raft_consensus.cc:2781] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:34.735359 31174 raft_consensus.cc:686] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [term 1 LEADER]: Becoming Leader. State: Replica: 522a624ecf05418ca34fc8be9e39621f, State: Running, Role: LEADER
I20260810 21:09:34.735428 31174 consensus_queue.cc:237] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } }
I20260810 21:09:34.735766 31156 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: TxnStatusTablet state changed. Reason: New leader 522a624ecf05418ca34fc8be9e39621f. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.735857 31156 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:09:34.736116 30217 catalog_manager.cc:5495] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f reported cstate change: term changed from 0 to 1, leader changed from <none> to 522a624ecf05418ca34fc8be9e39621f (127.27.117.67). New cstate: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } health_report { overall_health: HEALTHY } } }
I20260810 21:09:34.736228 31210 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 21:09:34.736349 31210 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 21:09:34.742197 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.739843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.27.117.67:33531)
0810 21:09:34.739899 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.739902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.739976 (+ 74us) server_negotiation.cc:407] Connection header received
0810 21:09:34.740067 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.740068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.740080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.740126 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:34.740243 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.740717 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.741474 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.741664 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.741748 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.741936 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:34.741976 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:34.742038 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.742107 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.742130 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:09:34.742213 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.739834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:33531 (local address 127.0.0.1:59348)
0810 21:09:34.739903 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.739916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.739996 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.740130 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.740131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.740133 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:09:34.740183 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.740185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.740817 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.740818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.741386 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.741387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.741751 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.741752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.741819 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.741852 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:09:34.742040 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:09:34.742051 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.742092 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.742145 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 21:09:34.743309 30744 raft_consensus.cc:1261] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Refusing update from remote peer 522a624ecf05418ca34fc8be9e39621f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:34.743316 30485 raft_consensus.cc:1261] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Refusing update from remote peer 522a624ecf05418ca34fc8be9e39621f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:34.743534 31174 consensus_queue.cc:1035] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:34.743655 31156 consensus_queue.cc:1035] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:34.744352 31152 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: TxnStatusTablet state changed. Reason: New leader 522a624ecf05418ca34fc8be9e39621f. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.744494 31152 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:34.744346 31150 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: TxnStatusTablet state changed. Reason: New leader 522a624ecf05418ca34fc8be9e39621f. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.744603 31150 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:34.745239 31174 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.745317 31174 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:09:34.745594 31162 mvcc.cc:205] Tried to move back new op lower bound from 7317078731746885632 to 7317078731716800512. Current Snapshot: MvccSnapshot[applied={T|T < 7317078731746885632}]
I20260810 21:09:34.745842 31150 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.745926 31150 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P bfc459a3720c4658a308c6762854fd4e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 21:09:34.747036 30838 tablet.cc:2369] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:34.752513 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.749881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:51463 (local address 127.27.117.67:33531)
0810 21:09:34.749923 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.749925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.750018 (+ 93us) server_negotiation.cc:407] Connection header received
0810 21:09:34.750081 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.750082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.750091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.750144 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:34.750311 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.750822 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.751890 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.752059 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.752111 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.752291 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:34.752313 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:34.752352 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.752433 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.752448 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:09:34.752588 31113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.749825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:33531 (local address 127.27.117.67:51463)
0810 21:09:34.749943 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:34.749953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:34.750036 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:34.750183 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:34.750185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:34.750186 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:09:34.750234 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.750236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.750907 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.750908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.751796 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:34.751798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.752137 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.752138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:34.752196 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.752212 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:09:34.752376 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:09:34.752378 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:09:34.752426 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 21:09:34.752470 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 21:09:34.753523 31174 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.753604 31174 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 522a624ecf05418ca34fc8be9e39621f: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:09:34.753821 31152 tablet_replica.cc:439] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "522a624ecf05418ca34fc8be9e39621f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 } } peers { permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } } peers { permanent_uuid: "522a624ecf05418ca34fc8be9e39621f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 33531 } } }
I20260810 21:09:34.753969 31152 tablet_replica.cc:442] T 83b2a626c6ce40e1b6c14f3b0003e96f P 8e975bfcc6234932b5bdbe1986ab2dbf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:34.755339 30744 raft_consensus.cc:1261] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Refusing update from remote peer 522a624ecf05418ca34fc8be9e39621f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:34.755368 30485 raft_consensus.cc:1261] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Refusing update from remote peer 522a624ecf05418ca34fc8be9e39621f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:34.755532 31153 consensus_queue.cc:1035] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc459a3720c4658a308c6762854fd4e" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:34.755606 31156 consensus_queue.cc:1035] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e975bfcc6234932b5bdbe1986ab2dbf" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:34.757196 31161 mvcc.cc:205] Tried to move back new op lower bound from 7317078731796623360 to 7317078731261157376. Current Snapshot: MvccSnapshot[applied={T|T < 7317078731796623360}]
I20260810 21:09:34.773734 30217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:34.773826 30217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:34.774482 30217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5073314ec5649c38006dc7f09291ad3: Sending DeleteTablet for 3 replicas of tablet 36e7e662f74140e398480281d964c38b
I20260810 21:09:34.774744 30451 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e7e662f74140e398480281d964c38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:38154
I20260810 21:09:34.774893 31181 tablet_replica.cc:330] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: stopping tablet replica
I20260810 21:09:34.774926 30972 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e7e662f74140e398480281d964c38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:59338
I20260810 21:09:34.774967 31181 raft_consensus.cc:2229] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:34.775004 31181 raft_consensus.cc:2258] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:34.775022 31179 tablet_replica.cc:330] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: stopping tablet replica
I20260810 21:09:34.775077 31179 raft_consensus.cc:2229] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:34.775224 31179 raft_consensus.cc:2258] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:34.775652 31179 ts_tablet_manager.cc:1905] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:34.775878 30710 tablet_service.cc:1514] Processing DeleteTablet for tablet 36e7e662f74140e398480281d964c38b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:52504
I20260810 21:09:34.776007 31177 tablet_replica.cc:330] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: stopping tablet replica
I20260810 21:09:34.776078 31177 raft_consensus.cc:2229] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:34.776113 31177 raft_consensus.cc:2258] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:34.776386 31177 ts_tablet_manager.cc:1905] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:34.777650 31179 ts_tablet_manager.cc:1918] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:34.777740 31179 log.cc:1196] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/36e7e662f74140e398480281d964c38b
I20260810 21:09:34.778003 31179 ts_tablet_manager.cc:1939] T 36e7e662f74140e398480281d964c38b P 522a624ecf05418ca34fc8be9e39621f: Deleting consensus metadata
I20260810 21:09:34.778067 31177 ts_tablet_manager.cc:1918] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:34.778131 31177 log.cc:1196] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/36e7e662f74140e398480281d964c38b
I20260810 21:09:34.778301 30196 catalog_manager.cc:4841] TS 522a624ecf05418ca34fc8be9e39621f (127.27.117.67:33531): tablet 36e7e662f74140e398480281d964c38b (table test-table [id=3bbbe46d72224c2db39df390f2d9c248]) successfully deleted
I20260810 21:09:34.778343 31177 ts_tablet_manager.cc:1939] T 36e7e662f74140e398480281d964c38b P bfc459a3720c4658a308c6762854fd4e: Deleting consensus metadata
I20260810 21:09:34.778616 30198 catalog_manager.cc:4841] TS bfc459a3720c4658a308c6762854fd4e (127.27.117.66:43921): tablet 36e7e662f74140e398480281d964c38b (table test-table [id=3bbbe46d72224c2db39df390f2d9c248]) successfully deleted
I20260810 21:09:34.778672 31181 ts_tablet_manager.cc:1905] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:34.780272 31181 ts_tablet_manager.cc:1918] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:34.780339 31181 log.cc:1196] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/36e7e662f74140e398480281d964c38b
I20260810 21:09:34.780545 31181 ts_tablet_manager.cc:1939] T 36e7e662f74140e398480281d964c38b P 8e975bfcc6234932b5bdbe1986ab2dbf: Deleting consensus metadata
I20260810 21:09:34.780893 30198 catalog_manager.cc:4841] TS 8e975bfcc6234932b5bdbe1986ab2dbf (127.27.117.65:35511): tablet 36e7e662f74140e398480281d964c38b (table test-table [id=3bbbe46d72224c2db39df390f2d9c248]) successfully deleted
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:34.790571 31135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.777290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.27.117.65:35511)
0810 21:09:34.777333 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.777335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.780933 (+ 3598us) server_negotiation.cc:407] Connection header received
0810 21:09:34.781046 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.781047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.781060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.781123 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.781789 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.782287 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.783326 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.783529 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.783591 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.787001 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.787012 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.787012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.787020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.789340 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.789720 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.789724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.789724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.789790 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.790012 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.790012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.790013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.790159 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.790319 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.790390 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.790469 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 21:09:34.790889 30548 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38164
I20260810 21:09:34.798300 30304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:34.791962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.27.117.126:41691)
0810 21:09:34.792019 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:34.792021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:34.792031 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:34.792099 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:34.792100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:34.792111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:34.792166 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:34.792693 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.793274 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.794146 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:34.794364 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:34.794438 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:34.795895 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:34.795897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:34.795897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:34.795903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:34.797247 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.797624 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.797624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.797625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.797675 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:34.797878 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:34.797879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:34.797879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:34.797978 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:34.798052 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:34.798184 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 21:09:34.798240 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 21:09:34.798559 30217 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37096
I20260810 21:09:34.799203 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30336
I20260810 21:09:34.825361 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30587
I20260810 21:09:34.834224 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30853
I20260810 21:09:34.845288 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30167
2026-08-10T21:09:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2507 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[31232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[31232](info): set up 2 sockets
Aug 10 21:09:34 dist-test-slave-2kcd krb5kdc[31232](info): commencing operation
krb5kdc: starting...
W20260810 21:09:36.137245 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:36.255587 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38523
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:36.337241 31270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:36.337445 31270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:36.337466 31270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:36.338544 31270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:36.338578 31270 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 21:09:36.338591 31270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:36.338604 31270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:36.338615 31270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:36.338627 31270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:36.340013 31270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38523
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:36.340194 31270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:36.340355 31270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:36.345059 31281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:36.345027 31276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:36.345034 31275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:36.345176 31270 server_base.cc:1034] running on GCE node
I20260810 21:09:36.345476 31270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:36.345788 31270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:36.346930 31270 hybrid_clock.cc:648] HybridClock initialized: now 1786396176346896 us; error 52 us; skew 500 ppm
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:36.349928 31270 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:36.350435 31270 webserver.cc:460] Webserver started at http://127.27.117.126:32903/ using document root <none> and password file <none>
I20260810 21:09:36.350708 31270 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:36.350785 31270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:36.350932 31270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:36.352519 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "4ebcba1d15c043a6ab68f68073e8d9f7"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "0acaa73322292a6d829f8310f33a3ee3"
server_key_iv: "ecb1226bef65959cdcab9bbbaebcea5a"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.353027 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "4ebcba1d15c043a6ab68f68073e8d9f7"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "0acaa73322292a6d829f8310f33a3ee3"
server_key_iv: "ecb1226bef65959cdcab9bbbaebcea5a"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.360934 31270 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.002s
I20260810 21:09:36.362040 31291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:36.362224 31270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:36.362339 31270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "4ebcba1d15c043a6ab68f68073e8d9f7"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "0acaa73322292a6d829f8310f33a3ee3"
server_key_iv: "ecb1226bef65959cdcab9bbbaebcea5a"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.362426 31270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:36.386735 31270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:36.389762 31270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:36.389943 31270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:36.396574 31270 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:38523
I20260810 21:09:36.396598 31413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:38523 every 8 connection(s)
I20260810 21:09:36.397177 31270 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:36.397940 31415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:36.400490 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31270
I20260810 21:09:36.400586 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:36.400673 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Bootstrap starting.
I20260810 21:09:36.400753 28117 external_mini_cluster.cc:1428] Setting key 20e08d1908030047a8b5a93ad91014c9
I20260810 21:09:36.401506 31415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:36.401857 31415 log.cc:824] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:36.402735 31415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: No bootstrap required, opened a new log
I20260810 21:09:36.405247 31415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38523 } }
I20260810 21:09:36.405459 31415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:36.405500 31415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebcba1d15c043a6ab68f68073e8d9f7, State: Initialized, Role: FOLLOWER
I20260810 21:09:36.405652 31415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38523 } }
I20260810 21:09:36.405756 31415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:36.405795 31415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:36.405859 31415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:36.406587 31415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38523 } }
I20260810 21:09:36.406701 31415 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebcba1d15c043a6ab68f68073e8d9f7; no voters:
I20260810 21:09:36.406896 31415 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:36.406949 31423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:36.407125 31423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4ebcba1d15c043a6ab68f68073e8d9f7, State: Running, Role: LEADER
I20260810 21:09:36.407184 31415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:36.407219 31423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38523 } }
I20260810 21:09:36.407925 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38523 } } }
I20260810 21:09:36.408000 31423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebcba1d15c043a6ab68f68073e8d9f7. Latest consensus state: current_term: 1 leader_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcba1d15c043a6ab68f68073e8d9f7" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38523 } } }
I20260810 21:09:36.408082 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:36.408097 31423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:36.408390 31440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:36.409127 31440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:36.414019 31440 catalog_manager.cc:1347] Generated new cluster ID: fab4b19579e24c48a661baede4893960
I20260810 21:09:36.414074 31440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:36.415925 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.401330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.27.117.126:38523)
0810 21:09:36.401536 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:36.401545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:36.401581 (+ 36us) server_negotiation.cc:407] Connection header received
0810 21:09:36.402228 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:36.402236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:36.402285 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:36.402469 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:36.404072 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.405659 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.406830 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.407494 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.407615 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.411630 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:36.411633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:36.411634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:36.411644 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:36.413749 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.414273 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.414277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.414277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.414412 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.414632 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.414633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.414633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.414938 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:36.415114 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:36.415574 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 21:09:36.415723 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260810 21:09:36.453272 31440 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:36.454080 31440 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:36.463012 31440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Generated new TSK 0
I20260810 21:09:36.463234 31440 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:36.501245 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38523
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:36.618878 31458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:36.619143 31458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:36.619184 31458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:36.620579 31458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:36.620639 31458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:36.620669 31458 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 21:09:36.620687 31458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:09:36.620725 31458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:36.622860 31458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:36.623106 31458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:36.623345 31458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:36.629123 31471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:36.629142 31467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:36.629289 31458 server_base.cc:1034] running on GCE node
W20260810 21:09:36.629439 31469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:36.629886 31458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:36.630134 31458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:36.631282 31458 hybrid_clock.cc:648] HybridClock initialized: now 1786396176631265 us; error 42 us; skew 500 ppm
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:36.634266 31458 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:36.634716 31458 webserver.cc:460] Webserver started at http://127.27.117.65:39051/ using document root <none> and password file <none>
I20260810 21:09:36.634965 31458 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:36.635035 31458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:36.635179 31458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:36.636215 31458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "79f05eed8616486b8410cc8303d5c011"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "f196cfbd1954aa75fe12160cd0a6dacd"
server_key_iv: "42fc95d45e6d1e9ad01fdc5027faabc5"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.636588 31458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "79f05eed8616486b8410cc8303d5c011"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "f196cfbd1954aa75fe12160cd0a6dacd"
server_key_iv: "42fc95d45e6d1e9ad01fdc5027faabc5"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.638109 31458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:36.638953 31480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:36.639213 31458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:36.639317 31458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "79f05eed8616486b8410cc8303d5c011"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "f196cfbd1954aa75fe12160cd0a6dacd"
server_key_iv: "42fc95d45e6d1e9ad01fdc5027faabc5"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.639391 31458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:36.648934 31458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:36.651249 31458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:36.651361 31458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:36.651882 31458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:36.651948 31458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:36.651994 31458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:36.652020 31458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:36.659850 31458 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:32933
I20260810 21:09:36.660048 31689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:32933 every 8 connection(s)
I20260810 21:09:36.660252 31458 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:36.663683 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.652473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:52773 (local address 127.27.117.126:38523)
0810 21:09:36.652524 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:36.652526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:36.652836 (+ 310us) server_negotiation.cc:407] Connection header received
0810 21:09:36.653524 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:36.653526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:36.653541 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:36.653597 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:36.654488 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.655258 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.656852 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.657197 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.657245 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.660293 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:36.660294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:36.660295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:36.660325 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:36.662000 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.662487 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.662488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.662488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.662554 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.662812 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.662813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.662813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.662999 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:36.663083 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:36.663438 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 21:09:36.663549 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:09:36.663806 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.652433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.27.117.65:52773)
0810 21:09:36.652630 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:36.652707 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:36.653350 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:36.653664 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:36.653669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:36.654062 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:36.654344 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.654351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.655408 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.655410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.656702 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.656705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.657405 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.657406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.657560 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.658150 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:36.658159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:36.660189 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:36.662091 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.662099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.662101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.662405 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.662611 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.662612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.662612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.662742 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.663082 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:36.663088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:36.663272 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 21:09:36.663405 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 21:09:36.663569 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260810 21:09:36.665565 31691 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38523
I20260810 21:09:36.665688 31691 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:36.665903 31691 heartbeater.cc:511] Master 127.27.117.126:38523 requested a full tablet report, sending...
I20260810 21:09:36.666499 31319 ts_manager.cc:194] Registered new tserver with Master: 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933)
I20260810 21:09:36.667492 31319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:52773
I20260810 21:09:36.668287 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31458
I20260810 21:09:36.668390 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:36.668648 28117 external_mini_cluster.cc:1428] Setting key dbbce597337e805fd4383c26fa8cf0e7
I20260810 21:09:36.676565 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.667816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.27.117.65:36551)
0810 21:09:36.668848 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:36.668862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:36.668921 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:36.670116 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:36.670117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:36.670338 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:36.670405 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.670407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.671331 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.671332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.671836 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.671838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.672283 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.672284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.672339 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.672829 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:36.672835 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:36.673238 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:36.675298 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.675299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.675299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.675553 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.675797 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.675798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.675798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.675885 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.676203 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:36.676204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:36.676258 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 21:09:36.676349 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 21:09:36.676458 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":75,"threads_started":1}
I20260810 21:09:36.676579 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.668691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:36551 (local address 127.27.117.126:38523)
0810 21:09:36.669954 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:36.669956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:36.669967 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:36.670004 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:36.670005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:36.670017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:36.670090 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:36.670502 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.671208 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.671910 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.672195 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.672287 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.673349 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:36.673350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:36.673351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:36.673358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:36.675179 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.675645 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.675646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.675646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.675711 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.675968 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.675969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.675969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.676116 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:36.676196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:36.676396 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 21:09:36.676477 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237}
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:36.697100 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38523
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:36.793082 31705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:36.793272 31705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:36.793296 31705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:36.794677 31705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:36.794747 31705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:36.794780 31705 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 21:09:36.794803 31705 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:09:36.794831 31705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:36.796414 31705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:36.796693 31705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:36.796959 31705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:36.801759 31715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:36.801816 31712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:36.801918 31705 server_base.cc:1034] running on GCE node
W20260810 21:09:36.802016 31719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:36.802230 31705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:36.802479 31705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:36.803642 31705 hybrid_clock.cc:648] HybridClock initialized: now 1786396176803617 us; error 41 us; skew 500 ppm
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:36.806656 31705 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:36.807180 31705 webserver.cc:460] Webserver started at http://127.27.117.66:42605/ using document root <none> and password file <none>
I20260810 21:09:36.807482 31705 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:36.807534 31705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:36.807667 31705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:36.808858 31705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "dd380163a44c540a51e8c257327a419b"
server_key_iv: "15e1357e68ade83d2262f3e5d32c5d31"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.809242 31705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "dd380163a44c540a51e8c257327a419b"
server_key_iv: "15e1357e68ade83d2262f3e5d32c5d31"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.810678 31705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:36.811520 31730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:36.811751 31705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:36.811875 31705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
format_stamp: "Formatted at 2026-08-10 21:09:36 on dist-test-slave-2kcd"
server_key: "dd380163a44c540a51e8c257327a419b"
server_key_iv: "15e1357e68ade83d2262f3e5d32c5d31"
server_key_version: "encryptionkey@0"
I20260810 21:09:36.811959 31705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:36.822258 31705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:36.824558 31705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:36.824726 31705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:36.825380 31705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:36.825435 31705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:36.825479 31705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:36.825507 31705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:09:36 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:36.833244 31705 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:39049
I20260810 21:09:36.833367 31939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:39049 every 8 connection(s)
I20260810 21:09:36.833640 31705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:36.836549 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.825892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:54903 (local address 127.27.117.126:38523)
0810 21:09:36.825960 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:36.825962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:36.826177 (+ 215us) server_negotiation.cc:407] Connection header received
0810 21:09:36.826664 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:36.826666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:36.826679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:36.826765 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:36.827634 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.828479 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.830194 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.830512 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.830594 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.833415 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:36.833417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:36.833417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:36.833426 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:36.835021 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.835529 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.835530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.835530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.835585 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.835904 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.835904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.835904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.836011 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:36.836069 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:36.836413 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 21:09:36.836483 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:09:36.836834 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.825874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.27.117.66:54903)
0810 21:09:36.826040 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:36.826081 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:36.826569 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:36.826859 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:36.826864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:36.827226 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:36.827494 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.827501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.828617 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.828619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.830105 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.830109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.830686 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.830687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.830844 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.831411 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:36.831422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:36.833301 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:36.835114 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.835121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.835124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.835443 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.835658 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.835659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.835660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.835819 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.836088 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:36.836095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:36.836262 (+ 167us) client_negotiation.cc:769] Sending connection context
0810 21:09:36.836391 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 21:09:36.836566 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260810 21:09:36.837534 31943 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38523
I20260810 21:09:36.837632 31943 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:36.837828 31943 heartbeater.cc:511] Master 127.27.117.126:38523 requested a full tablet report, sending...
I20260810 21:09:36.838567 31319 ts_manager.cc:194] Registered new tserver with Master: a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049)
I20260810 21:09:36.839087 31319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:54903
I20260810 21:09:36.842258 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31705
I20260810 21:09:36.842374 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:36.842588 28117 external_mini_cluster.cc:1428] Setting key f7122b498e667e207bc2e87d18506bb1
I20260810 21:09:36.845467 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.838669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:40621 (local address 127.27.117.126:38523)
0810 21:09:36.838712 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:36.838714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:36.838906 (+ 192us) server_negotiation.cc:407] Connection header received
0810 21:09:36.838954 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:36.838954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:36.838964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:36.839020 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:36.839428 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.840171 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.841238 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.841418 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.841475 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.842471 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:36.842472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:36.842472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:36.842480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:36.844093 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.844604 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.844605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.844605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.844664 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:36.844882 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:36.844883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:36.844883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:36.845036 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:36.845123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:36.845302 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 21:09:36.845401 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:09:36.845475 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:36.838676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.27.117.66:40621)
0810 21:09:36.838820 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:36.838832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:36.838901 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:36.839046 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:36.839047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:36.839261 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:36.839324 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.839326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.840294 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.840295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.841153 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:36.841156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:36.841501 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:36.841502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:36.841586 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:36.841976 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:36.841981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:36.842356 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:36.844210 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.844211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.844211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.844508 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.844744 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:36.844744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:36.844745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:36.844831 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:36.845113 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:36.845114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:36.845164 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 21:09:36.845245 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:09:36.845368 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:36.872421 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38523
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:37.021029 31957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:37.021322 31957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:37.021369 31957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:37.023104 31957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:37.023164 31957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:37.023195 31957 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 21:09:37.023211 31957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:09:37.023244 31957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:37.025605 31957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:38523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:37.025995 31957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:37.026309 31957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:37.032917 31970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:37.032850 31965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:37.033007 31964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:37.033190 31957 server_base.cc:1034] running on GCE node
I20260810 21:09:37.033390 31957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:37.033632 31957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:37.034795 31957 hybrid_clock.cc:648] HybridClock initialized: now 1786396177034768 us; error 44 us; skew 500 ppm
Aug 10 21:09:37 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396177, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:37.037604 31957 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:37.038293 31957 webserver.cc:460] Webserver started at http://127.27.117.67:45417/ using document root <none> and password file <none>
I20260810 21:09:37.038544 31957 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:37.038589 31957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:37.038719 31957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:37.040309 31957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "ea422423b1c04d13bbe0479311924847"
format_stamp: "Formatted at 2026-08-10 21:09:37 on dist-test-slave-2kcd"
server_key: "83432705fcca1b13dd73ac25bbe351ca"
server_key_iv: "6356a4285a0efff0fdc14c0781bbf8f7"
server_key_version: "encryptionkey@0"
I20260810 21:09:37.040980 31957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "ea422423b1c04d13bbe0479311924847"
format_stamp: "Formatted at 2026-08-10 21:09:37 on dist-test-slave-2kcd"
server_key: "83432705fcca1b13dd73ac25bbe351ca"
server_key_iv: "6356a4285a0efff0fdc14c0781bbf8f7"
server_key_version: "encryptionkey@0"
I20260810 21:09:37.043040 31957 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 21:09:37.044212 31982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:37.044405 31957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:37.044514 31957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "ea422423b1c04d13bbe0479311924847"
format_stamp: "Formatted at 2026-08-10 21:09:37 on dist-test-slave-2kcd"
server_key: "83432705fcca1b13dd73ac25bbe351ca"
server_key_iv: "6356a4285a0efff0fdc14c0781bbf8f7"
server_key_version: "encryptionkey@0"
I20260810 21:09:37.044606 31957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:37.071002 31957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:37.073571 31957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:37.073807 31957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:37.074739 31957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:37.074813 31957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:37.074862 31957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:37.074888 31957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:09:37 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396177, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:37.088647 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.075346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:47905 (local address 127.27.117.126:38523)
0810 21:09:37.075422 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.075425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.075676 (+ 251us) server_negotiation.cc:407] Connection header received
0810 21:09:37.076485 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.076486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.076508 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.076602 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:37.077628 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.078899 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.080625 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.080898 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.081006 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.084953 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:37.084954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:37.084955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:37.084963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:37.086915 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.087569 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.087570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.087570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.087627 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.087910 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.087911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.087911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.088043 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:37.088123 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.088441 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.088527 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 21:09:37.088896 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.075208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.27.117.67:47905)
0810 21:09:37.075548 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.075585 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.076307 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.076730 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.076735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.077078 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:37.077509 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.077515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.079049 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.079050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.080516 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.080519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.081080 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.081081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.081362 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.082101 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:37.082111 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:37.084827 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:37.087030 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:37.087039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:37.087042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:37.087476 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:37.087719 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:37.087720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:37.087720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:37.087841 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:37.088114 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:37.088121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:37.088290 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.088418 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.088601 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":175,"threads_started":1}
I20260810 21:09:37.094152 31957 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:35055
I20260810 21:09:37.094635 31957 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:37.094707 32216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:35055 every 8 connection(s)
I20260810 21:09:37.095531 32218 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38523
I20260810 21:09:37.095940 32218 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:37.096266 32218 heartbeater.cc:511] Master 127.27.117.126:38523 requested a full tablet report, sending...
I20260810 21:09:37.096834 31319 ts_manager.cc:194] Registered new tserver with Master: ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:37.097400 31319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:47905
I20260810 21:09:37.099061 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31957
I20260810 21:09:37.099161 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:37.099382 28117 external_mini_cluster.cc:1428] Setting key a9690d2fd6e03139f759860f91c97be0
I20260810 21:09:37.100347 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:09:37.100759 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.092054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:56145 (local address 127.27.117.126:38523)
0810 21:09:37.092114 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.092116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.092404 (+ 288us) server_negotiation.cc:407] Connection header received
0810 21:09:37.092479 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.092480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.092493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.092550 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:37.093448 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.094202 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.095833 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.096094 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.096175 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.097605 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:37.097607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:37.097607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:37.097615 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:37.099307 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.099734 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.099735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.099736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.099842 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.100079 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.100080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.100081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.100275 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:37.100354 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.100601 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.100680 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:09:37.100842 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.091933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.27.117.67:56145)
0810 21:09:37.092314 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.092326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.092411 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.092555 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.092556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.093013 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:37.093358 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.093361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.094466 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.094467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.095481 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.095483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.096174 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.096175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.096251 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.097063 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:37.097068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:37.097536 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:37.099410 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:37.099411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:37.099411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:37.099657 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:37.099920 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:37.099921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:37.099921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:37.099998 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:37.100354 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:37.100355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:37.100417 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.100548 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.100661 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":287,"threads_started":1}
Aug 10 21:09:37 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 21:09:37 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:37.131054 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.122549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.27.117.126:38523)
0810 21:09:37.122605 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.122617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.122736 (+ 119us) server_negotiation.cc:407] Connection header received
0810 21:09:37.122871 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.122874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.122888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.122955 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:37.123554 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.124390 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.125239 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.125603 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.125683 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.127976 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:37.127978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:37.127978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:37.127987 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:37.129690 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.130062 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.130063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.130064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.130154 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.130362 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.130363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.130363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.130524 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:37.130659 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.130745 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.130887 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:09:37.133241 31319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:37.136436 31319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:37.144490 32243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.139166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:39049 (local address 127.0.0.1:43228)
0810 21:09:37.139951 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.139962 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.140007 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.140396 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.140396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.140398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.140450 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.140451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.142022 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.142023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.143243 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.143246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.144243 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.144245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.144308 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.144312 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.144339 (+ 27us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.144416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260810 21:09:37.144757 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.139165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:35055 (local address 127.0.0.1:46144)
0810 21:09:37.139341 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.139384 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.139491 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.139956 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.139959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.139963 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.140037 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.140038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.141503 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.141505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.142913 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.142916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.144170 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.144171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.144274 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.144292 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.144345 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.144701 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":311,"thread_start_us":48,"threads_started":1}
I20260810 21:09:37.144860 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.139342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.27.117.67:35055)
0810 21:09:37.139693 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.139698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.139709 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:37.139749 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.139750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.139767 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.139844 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.140134 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.141377 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.143131 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.144059 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.144426 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.144492 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.144519 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.144563 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":190,"threads_started":1}
I20260810 21:09:37.145260 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.139717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.27.117.66:39049)
0810 21:09:37.140193 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.140197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.140208 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:37.140247 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.140250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.140265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.140524 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.140585 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.141926 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.143404 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.144154 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.145006 (+ 852us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.145071 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.145097 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.145135 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":118,"threads_started":1}
I20260810 21:09:37.145638 32098 tablet_service.cc:1467] Processing CreateTablet for tablet dde8224dee404020aa46921a606f62ed (DEFAULT_TABLE table=test-table [id=b0c24127e5dc444f8caa7124a7536246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:37.145962 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.139356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32933 (local address 127.0.0.1:33386)
0810 21:09:37.140078 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.140088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.140127 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.140452 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.140454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.140455 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.140488 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.140489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.142333 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.142334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.143595 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.143597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.145705 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.145706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.145770 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.145774 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.145836 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.145886 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":177,"threads_started":1}
I20260810 21:09:37.146090 32098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde8224dee404020aa46921a606f62ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.146042 31829 tablet_service.cc:1467] Processing CreateTablet for tablet dde8224dee404020aa46921a606f62ed (DEFAULT_TABLE table=test-table [id=b0c24127e5dc444f8caa7124a7536246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:37.146279 32244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.139505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.27.117.65:32933)
0810 21:09:37.140231 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.140235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.140244 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:37.140334 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.140335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.140351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.140554 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.140612 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.142245 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.143689 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.145599 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.145691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.145743 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.146136 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.146177 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":61,"threads_started":1}
I20260810 21:09:37.146456 31829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde8224dee404020aa46921a606f62ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.146929 31588 tablet_service.cc:1467] Processing CreateTablet for tablet dde8224dee404020aa46921a606f62ed (DEFAULT_TABLE table=test-table [id=b0c24127e5dc444f8caa7124a7536246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:37.147294 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde8224dee404020aa46921a606f62ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.150486 32247 tablet_bootstrap.cc:492] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Bootstrap starting.
I20260810 21:09:37.152127 32247 tablet_bootstrap.cc:654] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.152917 32247 log.cc:824] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:37.153075 32250 tablet_bootstrap.cc:492] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Bootstrap starting.
I20260810 21:09:37.154153 32247 tablet_bootstrap.cc:492] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: No bootstrap required, opened a new log
I20260810 21:09:37.154263 32247 ts_tablet_manager.cc:1397] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 21:09:37.154634 32250 tablet_bootstrap.cc:654] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.155567 32250 log.cc:824] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:37.156082 32252 tablet_bootstrap.cc:492] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Bootstrap starting.
I20260810 21:09:37.157282 32250 tablet_bootstrap.cc:492] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: No bootstrap required, opened a new log
I20260810 21:09:37.157375 32250 ts_tablet_manager.cc:1397] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 21:09:37.158284 32247 raft_consensus.cc:348] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.158542 32247 raft_consensus.cc:374] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.158597 32247 raft_consensus.cc:729] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea422423b1c04d13bbe0479311924847, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.158766 32247 consensus_queue.cc:260] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.159183 32218 heartbeater.cc:503] Master 127.27.117.126:38523 was elected leader, sending a full tablet report...
I20260810 21:09:37.159525 32247 ts_tablet_manager.cc:1428] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260810 21:09:37.160961 32250 raft_consensus.cc:348] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.161125 32250 raft_consensus.cc:374] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.161177 32250 raft_consensus.cc:729] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.161296 32250 consensus_queue.cc:260] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.161562 32250 ts_tablet_manager.cc:1428] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260810 21:09:37.161732 31691 heartbeater.cc:503] Master 127.27.117.126:38523 was elected leader, sending a full tablet report...
I20260810 21:09:37.162302 32252 tablet_bootstrap.cc:654] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.162951 32252 log.cc:824] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:37.163786 32252 tablet_bootstrap.cc:492] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: No bootstrap required, opened a new log
I20260810 21:09:37.163846 32252 ts_tablet_manager.cc:1397] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260810 21:09:37.166877 32252 raft_consensus.cc:348] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.167023 32252 raft_consensus.cc:374] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.167045 32252 raft_consensus.cc:729] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05f766cbbcd43d98ddf6dce9651c1b8, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.167172 32252 consensus_queue.cc:260] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.167527 32252 ts_tablet_manager.cc:1428] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260810 21:09:37.167598 31943 heartbeater.cc:503] Master 127.27.117.126:38523 was elected leader, sending a full tablet report...
I20260810 21:09:37.265056 32262 raft_consensus.cc:482] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:37.265237 32262 raft_consensus.cc:504] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.265763 32262 leader_election.cc:288] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049), ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:37.269579 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.266143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:47165 (local address 127.27.117.67:35055)
0810 21:09:37.266263 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.266265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.266275 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:37.266347 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.266348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.266359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.266446 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.266637 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.267524 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.268831 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.269178 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.269267 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.269277 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.269465 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.269508 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 21:09:37.269639 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.265964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:35055 (local address 127.27.117.65:47165)
0810 21:09:37.266192 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.266204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.266281 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.266472 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.266473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.266475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.266533 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.266535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.267613 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.267614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.268703 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.268705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.269267 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.269269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.269368 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.269372 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.269456 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.269499 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20260810 21:09:37.270385 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.266153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:60043 (local address 127.27.117.66:39049)
0810 21:09:37.266223 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.266225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.266292 (+ 67us) server_negotiation.cc:407] Connection header received
0810 21:09:37.266375 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.266375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.266383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.266455 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.266778 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.267273 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.268809 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.269601 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.269706 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.269716 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.270242 (+ 526us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.270289 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:09:37.270386 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.265985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:39049 (local address 127.27.117.65:60043)
0810 21:09:37.266202 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.266216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.266294 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.266504 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.266505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.266506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.266698 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.266700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.267364 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.267365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.268678 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.268681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.269977 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.269978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.270124 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.270130 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.270199 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.270231 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":135,"threads_started":1}
I20260810 21:09:37.270352 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde8224dee404020aa46921a606f62ed" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea422423b1c04d13bbe0479311924847" is_pre_election: true
I20260810 21:09:37.270515 32126 raft_consensus.cc:2454] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 0.
I20260810 21:09:37.270876 31483 leader_election.cc:302] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:37.271018 32262 raft_consensus.cc:2781] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:37.270970 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde8224dee404020aa46921a606f62ed" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" is_pre_election: true
I20260810 21:09:37.271070 32262 raft_consensus.cc:482] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:37.271087 32262 raft_consensus.cc:3037] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.271148 31863 raft_consensus.cc:2454] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 0.
I20260810 21:09:37.272119 32262 raft_consensus.cc:504] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.272421 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde8224dee404020aa46921a606f62ed" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
I20260810 21:09:37.272465 32262 leader_election.cc:288] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 election: Requested vote from peers a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049), ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:37.272511 31863 raft_consensus.cc:3037] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.272555 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde8224dee404020aa46921a606f62ed" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea422423b1c04d13bbe0479311924847"
I20260810 21:09:37.272640 32126 raft_consensus.cc:3037] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.273820 32126 raft_consensus.cc:2454] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 1.
I20260810 21:09:37.273820 31863 raft_consensus.cc:2454] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 1.
I20260810 21:09:37.274088 31484 leader_election.cc:302] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, a05f766cbbcd43d98ddf6dce9651c1b8; no voters:
I20260810 21:09:37.274257 32262 raft_consensus.cc:2781] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:37.274410 32262 raft_consensus.cc:686] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 1 LEADER]: Becoming Leader. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Running, Role: LEADER
I20260810 21:09:37.274502 32262 consensus_queue.cc:237] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } }
I20260810 21:09:37.275403 31319 catalog_manager.cc:5495] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79f05eed8616486b8410cc8303d5c011 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "79f05eed8616486b8410cc8303d5c011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } health_report { overall_health: HEALTHY } } }
Aug 10 21:09:37 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396176, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:37.318230 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.309934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38523 (local address 127.0.0.1:45340)
0810 21:09:37.310176 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.310196 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.310254 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.310416 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.310417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.310725 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:37.310772 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.310774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.311899 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.311900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.312412 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.312413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.313088 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.313089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.313134 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.313728 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:37.313733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:37.315258 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:37.317171 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:37.317178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:37.317179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:37.317382 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:37.317613 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:37.317613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:37.317613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:37.317667 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:37.317975 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:37.317980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:37.318018 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.318072 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.318160 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":184,"threads_started":1}
I20260810 21:09:37.318425 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.310013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.27.117.126:38523)
0810 21:09:37.310066 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.310068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.310265 (+ 197us) server_negotiation.cc:407] Connection header received
0810 21:09:37.310326 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.310327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.310339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.310404 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:37.310861 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.311808 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.312503 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.312944 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.313082 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.315355 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:37.315356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:37.315357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:37.315364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:37.317092 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.317468 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.317469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.317469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.317540 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:37.317732 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:37.317733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:37.317734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:37.317903 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:37.317983 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.318167 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.318325 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:09:37.320993 31319 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:45340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 21:09:37.321282 31319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:37.322981 31829 tablet_service.cc:1467] Processing CreateTablet for tablet a70b26acfbde4403997a2565b0cffd7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adecf37396b0432cbcf2894dbeecd598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:09:37.323019 31588 tablet_service.cc:1467] Processing CreateTablet for tablet a70b26acfbde4403997a2565b0cffd7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adecf37396b0432cbcf2894dbeecd598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:09:37.323024 32098 tablet_service.cc:1467] Processing CreateTablet for tablet a70b26acfbde4403997a2565b0cffd7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adecf37396b0432cbcf2894dbeecd598]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:09:37.323170 31829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70b26acfbde4403997a2565b0cffd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.323170 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70b26acfbde4403997a2565b0cffd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.323213 32098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70b26acfbde4403997a2565b0cffd7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.324723 32247 tablet_bootstrap.cc:492] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: Bootstrap starting.
I20260810 21:09:37.325083 32252 tablet_bootstrap.cc:492] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: Bootstrap starting.
I20260810 21:09:37.325404 32247 tablet_bootstrap.cc:654] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.326362 32252 tablet_bootstrap.cc:654] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.326514 32247 tablet_bootstrap.cc:492] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: No bootstrap required, opened a new log
I20260810 21:09:37.326571 32247 ts_tablet_manager.cc:1397] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:37.326807 32247 raft_consensus.cc:348] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.326881 32250 tablet_bootstrap.cc:492] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: Bootstrap starting.
I20260810 21:09:37.326916 32247 raft_consensus.cc:374] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.326948 32247 raft_consensus.cc:729] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea422423b1c04d13bbe0479311924847, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.327020 32247 consensus_queue.cc:260] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.327159 32247 ts_tablet_manager.cc:1428] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:37.327342 32260 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.327418 32260 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:37.327602 32250 tablet_bootstrap.cc:654] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.328409 32252 tablet_bootstrap.cc:492] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: No bootstrap required, opened a new log
I20260810 21:09:37.328469 32252 ts_tablet_manager.cc:1397] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 21:09:37.328938 32252 raft_consensus.cc:348] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.329046 32252 raft_consensus.cc:374] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.329063 32252 raft_consensus.cc:729] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05f766cbbcd43d98ddf6dce9651c1b8, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.329133 32252 consensus_queue.cc:260] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.329231 32252 ts_tablet_manager.cc:1428] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:37.329445 32250 tablet_bootstrap.cc:492] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: No bootstrap required, opened a new log
I20260810 21:09:37.329385 32266 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.329504 32266 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:37.329512 32250 ts_tablet_manager.cc:1397] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 21:09:37.330063 32250 raft_consensus.cc:348] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.330163 32250 raft_consensus.cc:374] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.330190 32250 raft_consensus.cc:729] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.330261 32250 consensus_queue.cc:260] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.330351 32250 ts_tablet_manager.cc:1428] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:37.330412 32262 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.330555 32262 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 21:09:37.334501 31944 tablet.cc:2369] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:09:37.345417 32220 tablet.cc:2369] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:09:37.411217 31692 tablet.cc:2369] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:37.429921 32260 raft_consensus.cc:482] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:37.430039 32260 raft_consensus.cc:504] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.430436 32260 leader_election.cc:288] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933), a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049)
I20260810 21:09:37.433506 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.430634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32933 (local address 127.27.117.67:59213)
0810 21:09:37.430701 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.430714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.430780 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.430988 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.430989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.430991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.431075 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.431077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.431763 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.431764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.432670 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.432672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.433257 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.433258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.433365 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.433368 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.433410 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.433443 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 21:09:37.433539 32244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.430756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:59213 (local address 127.27.117.65:32933)
0810 21:09:37.430818 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.430819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.430829 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:37.430870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.430871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.430881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.430961 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.431152 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.431680 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.432761 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.433178 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.433240 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.433247 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.433433 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.433470 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:09:37.433851 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b26acfbde4403997a2565b0cffd7c" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f05eed8616486b8410cc8303d5c011" is_pre_election: true
I20260810 21:09:37.434026 31621 raft_consensus.cc:2454] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 0.
I20260810 21:09:37.434271 31984 leader_election.cc:302] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:37.434379 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.430761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45515 (local address 127.27.117.66:39049)
0810 21:09:37.430820 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.430823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.430939 (+ 116us) server_negotiation.cc:407] Connection header received
0810 21:09:37.431061 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.431062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.431110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.431167 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:37.431304 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.432206 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.433336 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.433880 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.433954 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.433961 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.434280 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.434319 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 21:09:37.434497 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.430634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:39049 (local address 127.27.117.67:45515)
0810 21:09:37.430870 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.430879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.430955 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.431181 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.431181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.431183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:37.431235 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.431236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.432357 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.432358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.433256 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.433257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.433972 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.433973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.434141 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.434147 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.434268 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.434311 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":54,"thread_start_us":112,"threads_started":1}
I20260810 21:09:37.434553 32260 raft_consensus.cc:2781] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:37.434609 32260 raft_consensus.cc:482] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:37.434635 32260 raft_consensus.cc:3037] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.434818 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b26acfbde4403997a2565b0cffd7c" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" is_pre_election: true
I20260810 21:09:37.434923 31863 raft_consensus.cc:2454] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 0.
I20260810 21:09:37.436125 32260 raft_consensus.cc:504] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.436295 32260 leader_election.cc:288] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 election: Requested vote from peers 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933), a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049)
I20260810 21:09:37.436466 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b26acfbde4403997a2565b0cffd7c" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
I20260810 21:09:37.436565 31863 raft_consensus.cc:3037] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.436540 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70b26acfbde4403997a2565b0cffd7c" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f05eed8616486b8410cc8303d5c011"
I20260810 21:09:37.436659 31621 raft_consensus.cc:3037] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.437239 31863 raft_consensus.cc:2454] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 1.
I20260810 21:09:37.437489 31986 leader_election.cc:302] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05f766cbbcd43d98ddf6dce9651c1b8, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:37.437667 32260 raft_consensus.cc:2781] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:37.437963 32260 raft_consensus.cc:686] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [term 1 LEADER]: Becoming Leader. State: Replica: ea422423b1c04d13bbe0479311924847, State: Running, Role: LEADER
I20260810 21:09:37.438086 32260 consensus_queue.cc:237] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.438134 31621 raft_consensus.cc:2454] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 1.
I20260810 21:09:37.438807 32291 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: TxnStatusTablet state changed. Reason: New leader ea422423b1c04d13bbe0479311924847. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.438918 32291 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:09:37.439181 31319 catalog_manager.cc:5495] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea422423b1c04d13bbe0479311924847 (127.27.117.67). New cstate: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } health_report { overall_health: HEALTHY } } }
I20260810 21:09:37.439518 32294 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 21:09:37.439656 32294 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 21:09:37.473974 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.470129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:35055 (local address 127.0.0.1:46146)
0810 21:09:37.470223 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.470235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.470289 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.470788 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.470789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.470791 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:09:37.470860 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.470862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.471836 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.471837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.472711 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.472715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.473295 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.473296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.473392 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.473412 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.473806 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.473817 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.473871 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.473913 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 21:09:37.474020 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.470496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.27.117.67:35055)
0810 21:09:37.470569 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.470571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.470590 (+ 19us) server_negotiation.cc:407] Connection header received
0810 21:09:37.470636 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.470637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.470659 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.470741 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:37.470978 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.471737 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.472875 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.473200 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.473276 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.473534 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.473702 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.473808 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.473885 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.473908 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:09:37.476727 31621 raft_consensus.cc:1261] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Refusing update from remote peer ea422423b1c04d13bbe0479311924847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:37.476727 31863 raft_consensus.cc:1261] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Refusing update from remote peer ea422423b1c04d13bbe0479311924847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:37.477114 32291 consensus_queue.cc:1035] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:37.477242 32260 consensus_queue.cc:1035] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:37.478624 32266 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: TxnStatusTablet state changed. Reason: New leader ea422423b1c04d13bbe0479311924847. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.478713 32266 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:37.478642 32262 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: TxnStatusTablet state changed. Reason: New leader ea422423b1c04d13bbe0479311924847. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.478842 32262 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:37.480753 32262 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.480871 32262 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P 79f05eed8616486b8410cc8303d5c011: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:37.480821 32266 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.480914 32266 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P a05f766cbbcd43d98ddf6dce9651c1b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:09:37.481171 32302 mvcc.cc:205] Tried to move back new op lower bound from 7317078742941487104 to 7317078742787014656. Current Snapshot: MvccSnapshot[applied={T|T < 7317078742941487104}]
I20260810 21:09:37.481383 32300 mvcc.cc:205] Tried to move back new op lower bound from 7317078742941487104 to 7317078742787014656. Current Snapshot: MvccSnapshot[applied={T|T < 7317078742941487104}]
I20260810 21:09:37.481588 32301 mvcc.cc:205] Tried to move back new op lower bound from 7317078742941487104 to 7317078742787014656. Current Snapshot: MvccSnapshot[applied={T|T < 7317078742941487104 or (T in {7317078742941487104})}]
I20260810 21:09:37.481585 32291 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.481665 32291 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:09:37.481719 32291 tablet_replica.cc:439] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } } }
I20260810 21:09:37.481765 32291 tablet_replica.cc:442] T a70b26acfbde4403997a2565b0cffd7c P ea422423b1c04d13bbe0479311924847: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:09:37.487296 32244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.484056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.27.117.65:32933)
0810 21:09:37.484111 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.484113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.484127 (+ 14us) server_negotiation.cc:407] Connection header received
0810 21:09:37.484197 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.484198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.484216 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.484292 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:37.484474 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.485329 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.486361 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.486684 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.486758 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.486911 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.487028 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.487112 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.487193 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.487214 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:09:37.494179 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.490505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:35055 (local address 127.27.117.65:58779)
0810 21:09:37.490844 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.490866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.490973 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.491204 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.491205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.491207 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:09:37.491335 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.491336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.492053 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.492054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.493184 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.493187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.493541 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.493542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.493680 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.493697 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.493947 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.493949 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.494004 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.494049 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":160,"threads_started":1}
I20260810 21:09:37.494213 32241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.490763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:58779 (local address 127.27.117.67:35055)
0810 21:09:37.490816 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.490818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.490928 (+ 110us) server_negotiation.cc:407] Connection header received
0810 21:09:37.491074 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.491075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.491084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.491147 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:37.491416 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.491952 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.493271 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.493447 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.493506 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.493812 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.493847 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.493911 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.494028 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.494083 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 21:09:37.497877 32126 raft_consensus.cc:1261] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Refusing update from remote peer 79f05eed8616486b8410cc8303d5c011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:37.498090 31863 raft_consensus.cc:1261] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Refusing update from remote peer 79f05eed8616486b8410cc8303d5c011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:37.498251 32269 consensus_queue.cc:1035] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:37.498399 32262 consensus_queue.cc:1035] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:37.511159 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.507004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32933 (local address 127.27.117.67:44347)
0810 21:09:37.507102 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:37.507116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:37.507160 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:37.507469 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:37.507470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:37.507471 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:09:37.507625 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.507626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.508607 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.508608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.509555 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:37.509558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.510654 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.510655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:37.510776 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.510793 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.510999 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.511000 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 21:09:37.511039 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 21:09:37.511082 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 21:09:37.511457 32244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:37.507136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:44347 (local address 127.27.117.65:32933)
0810 21:09:37.507297 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:37.507299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:37.507306 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:09:37.507332 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:37.507333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:37.507342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:37.507384 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:37.507706 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.508483 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.510247 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:37.510558 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:37.510619 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:37.510899 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:37.510933 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:37.510977 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:37.511339 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 21:09:37.511370 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 21:09:37.524772 31321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:37.524935 31321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:37.525648 31321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Sending DeleteTablet for 3 replicas of tablet dde8224dee404020aa46921a606f62ed
I20260810 21:09:37.526064 31829 tablet_service.cc:1514] Processing DeleteTablet for tablet dde8224dee404020aa46921a606f62ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:37 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43228
I20260810 21:09:37.526216 31321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:37.526371 31321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:37.526527 32098 tablet_service.cc:1514] Processing DeleteTablet for tablet dde8224dee404020aa46921a606f62ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:37 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:46144
I20260810 21:09:37.526801 31588 tablet_service.cc:1514] Processing DeleteTablet for tablet dde8224dee404020aa46921a606f62ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:37 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33386
I20260810 21:09:37.526896 32338 tablet_replica.cc:330] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: stopping tablet replica
I20260810 21:09:37.527000 32338 raft_consensus.cc:2229] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:37.527048 32338 raft_consensus.cc:2258] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:37.527474 32338 ts_tablet_manager.cc:1905] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:37.527665 31829 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad34293057c44a5a22a137a524fbf2f (DEFAULT_TABLE table=test-table [id=fa1c69a9a0ab4c9e8cdfa715d3c05536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:37.527755 31588 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad34293057c44a5a22a137a524fbf2f (DEFAULT_TABLE table=test-table [id=fa1c69a9a0ab4c9e8cdfa715d3c05536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:37.527796 31829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad34293057c44a5a22a137a524fbf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.527902 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad34293057c44a5a22a137a524fbf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.528892 32098 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad34293057c44a5a22a137a524fbf2f (DEFAULT_TABLE table=test-table [id=fa1c69a9a0ab4c9e8cdfa715d3c05536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:37.528999 32098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad34293057c44a5a22a137a524fbf2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:37.529428 32252 tablet_bootstrap.cc:492] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Bootstrap starting.
I20260810 21:09:37.529422 32338 ts_tablet_manager.cc:1918] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:37.529486 32338 log.cc:1196] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/dde8224dee404020aa46921a606f62ed
I20260810 21:09:37.530055 32338 ts_tablet_manager.cc:1939] T dde8224dee404020aa46921a606f62ed P ea422423b1c04d13bbe0479311924847: Deleting consensus metadata
I20260810 21:09:37.530332 32252 tablet_bootstrap.cc:654] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.530853 32247 tablet_bootstrap.cc:492] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Bootstrap starting.
I20260810 21:09:37.531512 32247 tablet_bootstrap.cc:654] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.532472 31292 catalog_manager.cc:4841] TS ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055): tablet dde8224dee404020aa46921a606f62ed (table test-table [id=b0c24127e5dc444f8caa7124a7536246]) successfully deleted
I20260810 21:09:37.532841 32247 tablet_bootstrap.cc:492] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: No bootstrap required, opened a new log
I20260810 21:09:37.532888 32247 ts_tablet_manager.cc:1397] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:37.533214 32247 raft_consensus.cc:348] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.533279 32247 raft_consensus.cc:374] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.533300 32247 raft_consensus.cc:729] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea422423b1c04d13bbe0479311924847, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.533352 32247 consensus_queue.cc:260] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.533438 32247 ts_tablet_manager.cc:1428] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:37.533681 32250 tablet_bootstrap.cc:492] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Bootstrap starting.
I20260810 21:09:37.534399 32250 tablet_bootstrap.cc:654] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:37.535441 32250 tablet_bootstrap.cc:492] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: No bootstrap required, opened a new log
I20260810 21:09:37.535521 32250 ts_tablet_manager.cc:1397] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 21:09:37.535892 32250 raft_consensus.cc:348] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.535975 32250 raft_consensus.cc:374] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.536002 32250 raft_consensus.cc:729] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.536053 32250 consensus_queue.cc:260] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.536166 32250 ts_tablet_manager.cc:1428] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:37.536929 32337 tablet_replica.cc:330] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: stopping tablet replica
I20260810 21:09:37.537062 32337 raft_consensus.cc:2229] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:37.537139 32337 raft_consensus.cc:2258] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:37.538875 32252 tablet_bootstrap.cc:492] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: No bootstrap required, opened a new log
I20260810 21:09:37.538945 32252 ts_tablet_manager.cc:1397] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260810 21:09:37.539198 32252 raft_consensus.cc:348] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.539289 32252 raft_consensus.cc:374] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:37.539309 32252 raft_consensus.cc:729] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05f766cbbcd43d98ddf6dce9651c1b8, State: Initialized, Role: FOLLOWER
I20260810 21:09:37.539387 32252 consensus_queue.cc:260] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.539484 32252 ts_tablet_manager.cc:1428] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:37.541368 32337 ts_tablet_manager.cc:1905] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:37.543291 32339 tablet_replica.cc:330] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: stopping tablet replica
I20260810 21:09:37.543572 32339 raft_consensus.cc:2229] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:37.543740 32339 raft_consensus.cc:2258] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:37.544279 32339 ts_tablet_manager.cc:1905] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:37.545197 32337 ts_tablet_manager.cc:1918] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:37.545300 32337 log.cc:1196] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/dde8224dee404020aa46921a606f62ed
I20260810 21:09:37.545707 32337 ts_tablet_manager.cc:1939] T dde8224dee404020aa46921a606f62ed P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting consensus metadata
I20260810 21:09:37.546315 31294 catalog_manager.cc:4841] TS a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049): tablet dde8224dee404020aa46921a606f62ed (table test-table [id=b0c24127e5dc444f8caa7124a7536246]) successfully deleted
I20260810 21:09:37.550001 32339 ts_tablet_manager.cc:1918] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:37.550107 32339 log.cc:1196] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/dde8224dee404020aa46921a606f62ed
I20260810 21:09:37.550395 32339 ts_tablet_manager.cc:1939] T dde8224dee404020aa46921a606f62ed P 79f05eed8616486b8410cc8303d5c011: Deleting consensus metadata
I20260810 21:09:37.550887 31292 catalog_manager.cc:4841] TS 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933): tablet dde8224dee404020aa46921a606f62ed (table test-table [id=b0c24127e5dc444f8caa7124a7536246]) successfully deleted
I20260810 21:09:37.619560 32262 raft_consensus.cc:482] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:37.619669 32262 raft_consensus.cc:504] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.619824 32262 leader_election.cc:288] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049), ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:37.620222 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad34293057c44a5a22a137a524fbf2f" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea422423b1c04d13bbe0479311924847" is_pre_election: true
I20260810 21:09:37.620338 32126 raft_consensus.cc:2454] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 0.
I20260810 21:09:37.620576 31483 leader_election.cc:302] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:37.620985 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad34293057c44a5a22a137a524fbf2f" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" is_pre_election: true
I20260810 21:09:37.621109 31863 raft_consensus.cc:2454] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 0.
I20260810 21:09:37.621285 32262 raft_consensus.cc:2781] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:37.621333 32262 raft_consensus.cc:482] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:37.621356 32262 raft_consensus.cc:3037] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.622575 32262 raft_consensus.cc:504] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.622712 32262 leader_election.cc:288] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 election: Requested vote from peers a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049), ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:37.623024 31863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad34293057c44a5a22a137a524fbf2f" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
I20260810 21:09:37.623121 31863 raft_consensus.cc:3037] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.624075 31863 raft_consensus.cc:2454] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 1.
I20260810 21:09:37.624245 31484 leader_election.cc:302] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, a05f766cbbcd43d98ddf6dce9651c1b8; no voters:
I20260810 21:09:37.624490 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad34293057c44a5a22a137a524fbf2f" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea422423b1c04d13bbe0479311924847"
I20260810 21:09:37.624595 32126 raft_consensus.cc:3037] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:37.625902 32126 raft_consensus.cc:2454] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 1.
I20260810 21:09:37.626621 32262 raft_consensus.cc:2781] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:37.626736 32262 raft_consensus.cc:686] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 1 LEADER]: Becoming Leader. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Running, Role: LEADER
I20260810 21:09:37.626796 32262 consensus_queue.cc:237] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:37.627403 31319 catalog_manager.cc:5495] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79f05eed8616486b8410cc8303d5c011 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "79f05eed8616486b8410cc8303d5c011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } health_report { overall_health: UNKNOWN } } }
I20260810 21:09:37.657259 31863 raft_consensus.cc:1261] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Refusing update from remote peer 79f05eed8616486b8410cc8303d5c011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:37.657289 32126 raft_consensus.cc:1261] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Refusing update from remote peer 79f05eed8616486b8410cc8303d5c011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:37.657497 32262 consensus_queue.cc:1035] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:37.657713 32262 consensus_queue.cc:1035] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 21:09:37.664841 32302 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 21:09:39.193596 31319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:39.193742 31319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:39.194772 31319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Sending DeleteTablet for 3 replicas of tablet 7ad34293057c44a5a22a137a524fbf2f
I20260810 21:09:39.195096 31588 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad34293057c44a5a22a137a524fbf2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:39 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33386
I20260810 21:09:39.195166 31829 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad34293057c44a5a22a137a524fbf2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:39 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43228
I20260810 21:09:39.195381 32098 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ad34293057c44a5a22a137a524fbf2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:39 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:46144
I20260810 21:09:39.195415 32399 tablet_replica.cc:330] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: stopping tablet replica
I20260810 21:09:39.195497 32399 raft_consensus.cc:2229] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:39.195595 32399 raft_consensus.cc:2258] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:39.196014 32399 ts_tablet_manager.cc:1905] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:39.199113 32399 ts_tablet_manager.cc:1918] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 21:09:39.199195 32399 log.cc:1196] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/7ad34293057c44a5a22a137a524fbf2f
I20260810 21:09:39.199492 32399 ts_tablet_manager.cc:1939] T 7ad34293057c44a5a22a137a524fbf2f P 79f05eed8616486b8410cc8303d5c011: Deleting consensus metadata
I20260810 21:09:39.199908 31292 catalog_manager.cc:4841] TS 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933): tablet 7ad34293057c44a5a22a137a524fbf2f (table test-table [id=fa1c69a9a0ab4c9e8cdfa715d3c05536]) successfully deleted
I20260810 21:09:39.200927 32402 tablet_replica.cc:330] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: stopping tablet replica
I20260810 21:09:39.201004 32402 raft_consensus.cc:2229] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:39.201050 32402 raft_consensus.cc:2258] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:39.201164 32401 tablet_replica.cc:330] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: stopping tablet replica
I20260810 21:09:39.201233 32401 raft_consensus.cc:2229] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:39.201277 32401 raft_consensus.cc:2258] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:39.201478 32402 ts_tablet_manager.cc:1905] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:39.203487 32402 ts_tablet_manager.cc:1918] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 21:09:39.203555 32402 log.cc:1196] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/7ad34293057c44a5a22a137a524fbf2f
I20260810 21:09:39.203764 32402 ts_tablet_manager.cc:1939] T 7ad34293057c44a5a22a137a524fbf2f P ea422423b1c04d13bbe0479311924847: Deleting consensus metadata
I20260810 21:09:39.204142 31292 catalog_manager.cc:4841] TS ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055): tablet 7ad34293057c44a5a22a137a524fbf2f (table test-table [id=fa1c69a9a0ab4c9e8cdfa715d3c05536]) successfully deleted
I20260810 21:09:39.205058 32401 ts_tablet_manager.cc:1905] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:39.206979 32401 ts_tablet_manager.cc:1918] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 21:09:39.207059 32401 log.cc:1196] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/7ad34293057c44a5a22a137a524fbf2f
I20260810 21:09:39.207362 32401 ts_tablet_manager.cc:1939] T 7ad34293057c44a5a22a137a524fbf2f P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting consensus metadata
I20260810 21:09:39.207934 31294 catalog_manager.cc:4841] TS a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049): tablet 7ad34293057c44a5a22a137a524fbf2f (table test-table [id=fa1c69a9a0ab4c9e8cdfa715d3c05536]) successfully deleted
Aug 10 21:09:39 dist-test-slave-2kcd krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 21:09:39 dist-test-slave-2kcd krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:39.225968 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:39.216936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.27.117.126:38523)
0810 21:09:39.217074 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:39.217076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:39.217154 (+ 78us) server_negotiation.cc:407] Connection header received
0810 21:09:39.217202 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:39.217202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:39.217216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:39.217348 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:39.218016 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.218567 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.219780 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.220162 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.220232 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:39.222659 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:39.222661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:39.222662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:39.222669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:39.224732 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:39.225160 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:39.225172 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:39.225173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:39.225237 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:39.225454 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:39.225455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:39.225456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:39.225576 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:39.225663 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:39.225791 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 21:09:39.225878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260810 21:09:39.227572 31319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:39.227715 31319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:39.229318 31829 tablet_service.cc:1467] Processing CreateTablet for tablet c4897a23c1134e4ca136c757708ec299 (DEFAULT_TABLE table=test-table [id=a56f3774aff84fe6a7c63900af8e29b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:39.229295 31588 tablet_service.cc:1467] Processing CreateTablet for tablet c4897a23c1134e4ca136c757708ec299 (DEFAULT_TABLE table=test-table [id=a56f3774aff84fe6a7c63900af8e29b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:39.229477 31829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4897a23c1134e4ca136c757708ec299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:39.229590 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4897a23c1134e4ca136c757708ec299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:39.230919 32416 tablet_bootstrap.cc:492] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Bootstrap starting.
I20260810 21:09:39.230965 32098 tablet_service.cc:1467] Processing CreateTablet for tablet c4897a23c1134e4ca136c757708ec299 (DEFAULT_TABLE table=test-table [id=a56f3774aff84fe6a7c63900af8e29b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:39.231105 32098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4897a23c1134e4ca136c757708ec299. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:39.231403 32416 tablet_bootstrap.cc:654] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:39.231771 32418 tablet_bootstrap.cc:492] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Bootstrap starting.
I20260810 21:09:39.232125 32416 tablet_bootstrap.cc:492] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: No bootstrap required, opened a new log
I20260810 21:09:39.232196 32416 ts_tablet_manager.cc:1397] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:39.232254 32418 tablet_bootstrap.cc:654] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:39.232376 32416 raft_consensus.cc:348] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.232470 32416 raft_consensus.cc:374] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:39.232506 32416 raft_consensus.cc:729] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Initialized, Role: FOLLOWER
I20260810 21:09:39.232553 32416 consensus_queue.cc:260] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.232659 32416 ts_tablet_manager.cc:1428] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:39.233388 32418 tablet_bootstrap.cc:492] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: No bootstrap required, opened a new log
I20260810 21:09:39.233453 32418 ts_tablet_manager.cc:1397] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:39.233672 32418 raft_consensus.cc:348] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.233757 32418 raft_consensus.cc:374] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:39.233783 32418 raft_consensus.cc:729] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05f766cbbcd43d98ddf6dce9651c1b8, State: Initialized, Role: FOLLOWER
I20260810 21:09:39.233836 32418 consensus_queue.cc:260] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.234042 32418 ts_tablet_manager.cc:1428] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:39.237260 32423 tablet_bootstrap.cc:492] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Bootstrap starting.
I20260810 21:09:39.237864 32423 tablet_bootstrap.cc:654] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:39.238605 32423 tablet_bootstrap.cc:492] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: No bootstrap required, opened a new log
I20260810 21:09:39.238662 32423 ts_tablet_manager.cc:1397] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:39.238857 32423 raft_consensus.cc:348] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.238942 32423 raft_consensus.cc:374] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:39.238963 32423 raft_consensus.cc:729] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea422423b1c04d13bbe0479311924847, State: Initialized, Role: FOLLOWER
I20260810 21:09:39.239013 32423 consensus_queue.cc:260] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.239125 32423 ts_tablet_manager.cc:1428] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:39.482923 32374 raft_consensus.cc:482] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:39.483035 32374 raft_consensus.cc:504] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.483199 32374 leader_election.cc:288] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933), a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049)
I20260810 21:09:39.483507 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4897a23c1134e4ca136c757708ec299" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" is_pre_election: true
I20260810 21:09:39.483510 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4897a23c1134e4ca136c757708ec299" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f05eed8616486b8410cc8303d5c011" is_pre_election: true
I20260810 21:09:39.483616 31861 raft_consensus.cc:2454] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 0.
I20260810 21:09:39.483636 31621 raft_consensus.cc:2454] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 0.
I20260810 21:09:39.483826 31984 leader_election.cc:302] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:39.483948 32374 raft_consensus.cc:2781] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:39.484004 32374 raft_consensus.cc:482] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:39.484027 32374 raft_consensus.cc:3037] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:39.484882 32374 raft_consensus.cc:504] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.485023 32374 leader_election.cc:288] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 election: Requested vote from peers 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933), a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049)
I20260810 21:09:39.485157 31621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4897a23c1134e4ca136c757708ec299" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79f05eed8616486b8410cc8303d5c011"
I20260810 21:09:39.485174 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4897a23c1134e4ca136c757708ec299" candidate_uuid: "ea422423b1c04d13bbe0479311924847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
I20260810 21:09:39.485244 31621 raft_consensus.cc:3037] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:39.485245 31861 raft_consensus.cc:3037] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:39.486025 31861 raft_consensus.cc:2454] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 1.
I20260810 21:09:39.486182 31621 raft_consensus.cc:2454] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea422423b1c04d13bbe0479311924847 in term 1.
I20260810 21:09:39.486186 31986 leader_election.cc:302] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a05f766cbbcd43d98ddf6dce9651c1b8, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:39.486320 32374 raft_consensus.cc:2781] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:39.486369 32374 raft_consensus.cc:686] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 1 LEADER]: Becoming Leader. State: Replica: ea422423b1c04d13bbe0479311924847, State: Running, Role: LEADER
I20260810 21:09:39.486413 32374 consensus_queue.cc:237] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.487181 31321 catalog_manager.cc:5495] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea422423b1c04d13bbe0479311924847 (127.27.117.67). New cstate: current_term: 1 leader_uuid: "ea422423b1c04d13bbe0479311924847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } health_report { overall_health: HEALTHY } } }
I20260810 21:09:39.521191 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:39.518155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.27.117.67:35055)
0810 21:09:39.518433 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:39.518436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:39.518445 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:39.518468 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:39.518469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:39.518482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:39.518571 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:39.518689 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.519564 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.520297 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.520681 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.520747 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:39.520912 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:39.520948 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:39.521000 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:39.521055 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 21:09:39.521079 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":163,"threads_started":1}
I20260810 21:09:39.526605 32442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:39.523572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:35055 (local address 127.27.117.67:60653)
0810 21:09:39.523798 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:39.523807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:39.523862 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:39.524003 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:39.524004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:39.524006 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:09:39.524075 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:39.524077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.525121 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.525122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:39.525673 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:39.525675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.526198 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.526199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:39.526287 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:39.526291 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:09:39.526468 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:09:39.526470 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:09:39.526501 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 21:09:39.526538 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":26,"thread_start_us":147,"threads_started":1}
I20260810 21:09:39.526691 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:39.523676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:60653 (local address 127.27.117.67:35055)
0810 21:09:39.523736 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:39.523738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:39.523867 (+ 129us) server_negotiation.cc:407] Connection header received
0810 21:09:39.523921 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:39.523921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:39.523930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:39.524000 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:39.524160 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.525046 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.525765 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.526127 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.526191 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:39.526373 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:39.526405 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:39.526456 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:39.526516 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 21:09:39.526539 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":39}
I20260810 21:09:39.528451 31861 raft_consensus.cc:1261] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Refusing update from remote peer ea422423b1c04d13bbe0479311924847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:39.528470 31621 raft_consensus.cc:1261] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Refusing update from remote peer ea422423b1c04d13bbe0479311924847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:39.528676 32361 consensus_queue.cc:1035] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:39.528890 32361 consensus_queue.cc:1035] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:39.543359 31321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:39.543466 31321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:39.544178 31321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Sending DeleteTablet for 3 replicas of tablet c4897a23c1134e4ca136c757708ec299
I20260810 21:09:39.544548 32096 tablet_service.cc:1514] Processing DeleteTablet for tablet c4897a23c1134e4ca136c757708ec299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:39 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:46144
I20260810 21:09:39.544842 32402 tablet_replica.cc:330] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: stopping tablet replica
I20260810 21:09:39.544916 32402 raft_consensus.cc:2229] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:39.544879 31321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 21:09:39.545002 32402 raft_consensus.cc:2258] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 21:09:39.545012 31321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:39.545086 31588 tablet_service.cc:1514] Processing DeleteTablet for tablet c4897a23c1134e4ca136c757708ec299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:39 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33386
I20260810 21:09:39.545185 32399 tablet_replica.cc:330] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: stopping tablet replica
I20260810 21:09:39.545255 32399 raft_consensus.cc:2229] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:39.545291 32399 raft_consensus.cc:2258] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:39.545646 31829 tablet_service.cc:1514] Processing DeleteTablet for tablet c4897a23c1134e4ca136c757708ec299 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:39 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43228
I20260810 21:09:39.545747 32401 tablet_replica.cc:330] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: stopping tablet replica
I20260810 21:09:39.545811 32401 raft_consensus.cc:2229] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:39.545849 32402 ts_tablet_manager.cc:1905] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:39.545866 32401 raft_consensus.cc:2258] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:39.547423 32402 ts_tablet_manager.cc:1918] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:39.547487 32402 log.cc:1196] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/c4897a23c1134e4ca136c757708ec299
I20260810 21:09:39.547744 32402 ts_tablet_manager.cc:1939] T c4897a23c1134e4ca136c757708ec299 P ea422423b1c04d13bbe0479311924847: Deleting consensus metadata
I20260810 21:09:39.548080 31292 catalog_manager.cc:4841] TS ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055): tablet c4897a23c1134e4ca136c757708ec299 (table test-table [id=a56f3774aff84fe6a7c63900af8e29b5]) successfully deleted
I20260810 21:09:39.549018 32096 tablet_service.cc:1467] Processing CreateTablet for tablet d6b5ef234bf8489ea17f953d80ced21b (DEFAULT_TABLE table=test-table [id=07b6c784590744cb9d1292e4e99b6e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:39.549134 32096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b5ef234bf8489ea17f953d80ced21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:39.549021 31588 tablet_service.cc:1467] Processing CreateTablet for tablet d6b5ef234bf8489ea17f953d80ced21b (DEFAULT_TABLE table=test-table [id=07b6c784590744cb9d1292e4e99b6e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:39.549286 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b5ef234bf8489ea17f953d80ced21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:39.550484 31829 tablet_service.cc:1467] Processing CreateTablet for tablet d6b5ef234bf8489ea17f953d80ced21b (DEFAULT_TABLE table=test-table [id=07b6c784590744cb9d1292e4e99b6e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:39.550590 31829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b5ef234bf8489ea17f953d80ced21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:39.550825 32416 tablet_bootstrap.cc:492] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011: Bootstrap starting.
I20260810 21:09:39.551311 32416 tablet_bootstrap.cc:654] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:39.551494 32423 tablet_bootstrap.cc:492] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Bootstrap starting.
I20260810 21:09:39.551821 32423 tablet_bootstrap.cc:654] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:39.551976 32416 tablet_bootstrap.cc:492] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011: No bootstrap required, opened a new log
I20260810 21:09:39.552021 32416 ts_tablet_manager.cc:1397] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:39.552021 32418 tablet_bootstrap.cc:492] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Bootstrap starting.
I20260810 21:09:39.552136 32416 raft_consensus.cc:348] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.552215 32416 raft_consensus.cc:374] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:39.552238 32416 raft_consensus.cc:729] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Initialized, Role: FOLLOWER
I20260810 21:09:39.552287 32416 consensus_queue.cc:260] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.552405 32416 ts_tablet_manager.cc:1428] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:39.552465 32423 tablet_bootstrap.cc:492] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: No bootstrap required, opened a new log
I20260810 21:09:39.552510 32423 ts_tablet_manager.cc:1397] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:39.552492 32399 ts_tablet_manager.cc:1905] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:39.552631 32423 raft_consensus.cc:348] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.552693 32423 raft_consensus.cc:374] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:39.552711 32423 raft_consensus.cc:729] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea422423b1c04d13bbe0479311924847, State: Initialized, Role: FOLLOWER
I20260810 21:09:39.552750 32423 consensus_queue.cc:260] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.552894 32423 ts_tablet_manager.cc:1428] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:39.552911 32418 tablet_bootstrap.cc:654] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:39.553807 32418 tablet_bootstrap.cc:492] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: No bootstrap required, opened a new log
I20260810 21:09:39.553860 32418 ts_tablet_manager.cc:1397] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 21:09:39.554045 32418 raft_consensus.cc:348] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.554141 32418 raft_consensus.cc:374] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:39.554167 32418 raft_consensus.cc:729] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05f766cbbcd43d98ddf6dce9651c1b8, State: Initialized, Role: FOLLOWER
I20260810 21:09:39.554222 32418 consensus_queue.cc:260] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.554323 32399 ts_tablet_manager.cc:1918] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:39.554349 32418 ts_tablet_manager.cc:1428] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:39.554379 32399 log.cc:1196] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/c4897a23c1134e4ca136c757708ec299
I20260810 21:09:39.554605 32399 ts_tablet_manager.cc:1939] T c4897a23c1134e4ca136c757708ec299 P 79f05eed8616486b8410cc8303d5c011: Deleting consensus metadata
I20260810 21:09:39.554914 31292 catalog_manager.cc:4841] TS 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933): tablet c4897a23c1134e4ca136c757708ec299 (table test-table [id=a56f3774aff84fe6a7c63900af8e29b5]) successfully deleted
I20260810 21:09:39.555701 32401 ts_tablet_manager.cc:1905] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:39.557379 32401 ts_tablet_manager.cc:1918] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:09:39.557428 32401 log.cc:1196] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/c4897a23c1134e4ca136c757708ec299
I20260810 21:09:39.557636 32401 ts_tablet_manager.cc:1939] T c4897a23c1134e4ca136c757708ec299 P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting consensus metadata
I20260810 21:09:39.557936 31294 catalog_manager.cc:4841] TS a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049): tablet c4897a23c1134e4ca136c757708ec299 (table test-table [id=a56f3774aff84fe6a7c63900af8e29b5]) successfully deleted
I20260810 21:09:39.891023 32391 raft_consensus.cc:482] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:39.891150 32391 raft_consensus.cc:504] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.891348 32391 leader_election.cc:288] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049), ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:39.891575 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b5ef234bf8489ea17f953d80ced21b" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea422423b1c04d13bbe0479311924847" is_pre_election: true
I20260810 21:09:39.891577 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b5ef234bf8489ea17f953d80ced21b" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" is_pre_election: true
I20260810 21:09:39.891664 32126 raft_consensus.cc:2454] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 0.
I20260810 21:09:39.891705 31861 raft_consensus.cc:2454] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 0.
I20260810 21:09:39.891810 31483 leader_election.cc:302] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, ea422423b1c04d13bbe0479311924847; no voters:
I20260810 21:09:39.891881 32391 raft_consensus.cc:2781] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:39.891907 32391 raft_consensus.cc:482] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:39.891929 32391 raft_consensus.cc:3037] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:39.893162 32391 raft_consensus.cc:504] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.893318 32391 leader_election.cc:288] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 election: Requested vote from peers a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049), ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055)
I20260810 21:09:39.893458 31861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b5ef234bf8489ea17f953d80ced21b" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8"
I20260810 21:09:39.893532 31861 raft_consensus.cc:3037] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:39.893494 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b5ef234bf8489ea17f953d80ced21b" candidate_uuid: "79f05eed8616486b8410cc8303d5c011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea422423b1c04d13bbe0479311924847"
I20260810 21:09:39.893564 32126 raft_consensus.cc:3037] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:39.894277 31861 raft_consensus.cc:2454] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 1.
I20260810 21:09:39.894450 32126 raft_consensus.cc:2454] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79f05eed8616486b8410cc8303d5c011 in term 1.
I20260810 21:09:39.894469 31484 leader_election.cc:302] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79f05eed8616486b8410cc8303d5c011, a05f766cbbcd43d98ddf6dce9651c1b8; no voters:
I20260810 21:09:39.894579 32391 raft_consensus.cc:2781] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:39.894712 32391 raft_consensus.cc:686] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [term 1 LEADER]: Becoming Leader. State: Replica: 79f05eed8616486b8410cc8303d5c011, State: Running, Role: LEADER
I20260810 21:09:39.894778 32391 consensus_queue.cc:237] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } }
I20260810 21:09:39.895491 31321 catalog_manager.cc:5495] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79f05eed8616486b8410cc8303d5c011 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "79f05eed8616486b8410cc8303d5c011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79f05eed8616486b8410cc8303d5c011" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 } health_report { overall_health: UNKNOWN } } }
I20260810 21:09:39.915750 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:39.913133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.27.117.65:32933)
0810 21:09:39.913281 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:39.913283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:39.913289 (+ 6us) server_negotiation.cc:407] Connection header received
0810 21:09:39.913307 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:39.913308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:39.913317 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:39.913386 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:39.913516 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.914049 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.914949 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:39.915225 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:39.915300 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:39.915461 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:39.915496 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:39.915553 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:39.915619 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 21:09:39.915641 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260810 21:09:39.918432 31863 raft_consensus.cc:1261] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Refusing update from remote peer 79f05eed8616486b8410cc8303d5c011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:39.918432 32126 raft_consensus.cc:1261] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Refusing update from remote peer 79f05eed8616486b8410cc8303d5c011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:39.918747 32455 consensus_queue.cc:1035] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea422423b1c04d13bbe0479311924847" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:39.918864 32391 consensus_queue.cc:1035] T d6b5ef234bf8489ea17f953d80ced21b P 79f05eed8616486b8410cc8303d5c011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05f766cbbcd43d98ddf6dce9651c1b8" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 39049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 21:09:39.924285 32449 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 21:09:41.450373 31319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:41.450493 31319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:41.451362 31319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebcba1d15c043a6ab68f68073e8d9f7: Sending DeleteTablet for 3 replicas of tablet d6b5ef234bf8489ea17f953d80ced21b
I20260810 21:09:41.451632 31829 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b5ef234bf8489ea17f953d80ced21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:41 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43228
I20260810 21:09:41.451638 32096 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b5ef234bf8489ea17f953d80ced21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:41 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:46144
I20260810 21:09:41.451675 31588 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b5ef234bf8489ea17f953d80ced21b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:41 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33386
I20260810 21:09:41.451836 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31458
I20260810 21:09:41.458566 32513 tablet_replica.cc:330] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: stopping tablet replica
I20260810 21:09:41.458691 32513 raft_consensus.cc:2229] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:41.458808 32513 raft_consensus.cc:2258] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:41.460007 32513 ts_tablet_manager.cc:1905] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:41.462809 32513 ts_tablet_manager.cc:1918] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 21:09:41.462941 32513 log.cc:1196] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/d6b5ef234bf8489ea17f953d80ced21b
I20260810 21:09:41.462961 32514 tablet_replica.cc:330] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: stopping tablet replica
I20260810 21:09:41.463027 32514 raft_consensus.cc:2229] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:41.463075 32514 raft_consensus.cc:2258] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:41.463356 32513 ts_tablet_manager.cc:1939] T d6b5ef234bf8489ea17f953d80ced21b P a05f766cbbcd43d98ddf6dce9651c1b8: Deleting consensus metadata
I20260810 21:09:41.463460 32514 ts_tablet_manager.cc:1905] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:41.463804 31294 catalog_manager.cc:4841] TS a05f766cbbcd43d98ddf6dce9651c1b8 (127.27.117.66:39049): tablet d6b5ef234bf8489ea17f953d80ced21b (table test-table [id=07b6c784590744cb9d1292e4e99b6e3c]) successfully deleted
I20260810 21:09:41.464880 32514 ts_tablet_manager.cc:1918] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 21:09:41.464953 32514 log.cc:1196] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/d6b5ef234bf8489ea17f953d80ced21b
I20260810 21:09:41.465164 32514 ts_tablet_manager.cc:1939] T d6b5ef234bf8489ea17f953d80ced21b P ea422423b1c04d13bbe0479311924847: Deleting consensus metadata
I20260810 21:09:41.465502 31292 catalog_manager.cc:4841] TS ea422423b1c04d13bbe0479311924847 (127.27.117.67:35055): tablet d6b5ef234bf8489ea17f953d80ced21b (table test-table [id=07b6c784590744cb9d1292e4e99b6e3c]) successfully deleted
W20260810 21:09:41.469151 31292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:32933 (error 108)
I20260810 21:09:41.469424 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31705
I20260810 21:09:41.475602 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:41.469555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32933 (local address 127.0.0.1:53820)
0810 21:09:41.475514 (+ 5959us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:41.475566 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:32933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5927,"thread_start_us":58,"threads_started":1}
W20260810 21:09:41.475719 31292 catalog_manager.cc:4584] TS 79f05eed8616486b8410cc8303d5c011 (127.27.117.65:32933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6b5ef234bf8489ea17f953d80ced21b: Network error: Client connection negotiation failed: client connection to 127.27.117.65:32933: connect: Connection refused (error 111)
I20260810 21:09:41.481709 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31957
I20260810 21:09:41.494855 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31270
2026-08-10T21:09:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6650 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:41 dist-test-slave-2kcd krb5kdc[32523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:41 dist-test-slave-2kcd krb5kdc[32523](info): set up 2 sockets
Aug 10 21:09:41 dist-test-slave-2kcd krb5kdc[32523](info): commencing operation
krb5kdc: starting...
W20260810 21:09:42.798452 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:42 dist-test-slave-2kcd krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:42.931556 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35697
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:43.012714 32541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:43.012974 32541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:43.013011 32541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:43.014142 32541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:43.014215 32541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:43.014240 32541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:43.014264 32541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:43.014292 32541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:43.015719 32541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:43.015985 32541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:43.016208 32541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:43.021139 32547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.021157 32548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.021258 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:43.021209 32541 server_base.cc:1034] running on GCE node
I20260810 21:09:43.021601 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:43.021880 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:43.023046 32541 hybrid_clock.cc:648] HybridClock initialized: now 1786396183023017 us; error 47 us; skew 500 ppm
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:43.025595 32541 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:43.026077 32541 webserver.cc:460] Webserver started at http://127.27.117.126:33461/ using document root <none> and password file <none>
I20260810 21:09:43.026302 32541 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:43.026346 32541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:43.026438 32541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:43.027406 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "f9a3bf7053414d89bf79781ed7f0149d"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "173c59bdd7eccfc1493fc409b6b13763"
server_key_iv: "1dbe08ae9118a2a77031aa1d644aa155"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.027701 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "f9a3bf7053414d89bf79781ed7f0149d"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "173c59bdd7eccfc1493fc409b6b13763"
server_key_iv: "1dbe08ae9118a2a77031aa1d644aa155"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.029001 32541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:43.029681 32562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.029861 32541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:43.029953 32541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "f9a3bf7053414d89bf79781ed7f0149d"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "173c59bdd7eccfc1493fc409b6b13763"
server_key_iv: "1dbe08ae9118a2a77031aa1d644aa155"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.030007 32541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:43.046952 32541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:43.048718 32541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:43.048897 32541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:43.052603 32541 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35697
I20260810 21:09:43.052677 32648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35697 every 8 connection(s)
I20260810 21:09:43.053033 32541 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:43.053905 32652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:43.056118 32652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d: Bootstrap starting.
I20260810 21:09:43.056643 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 32541
I20260810 21:09:43.056746 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:43.056887 32652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:43.056998 28117 external_mini_cluster.cc:1428] Setting key 3d167397fdc6e5eb6315ee239c9b1d49
I20260810 21:09:43.057233 32652 log.cc:824] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:43.058027 32652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d: No bootstrap required, opened a new log
I20260810 21:09:43.059633 32652 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a3bf7053414d89bf79781ed7f0149d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35697 } }
I20260810 21:09:43.059764 32652 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:43.059811 32652 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a3bf7053414d89bf79781ed7f0149d, State: Initialized, Role: FOLLOWER
I20260810 21:09:43.059945 32652 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a3bf7053414d89bf79781ed7f0149d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35697 } }
I20260810 21:09:43.060022 32652 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:43.060045 32652 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:43.060113 32652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:43.060971 32652 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a3bf7053414d89bf79781ed7f0149d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35697 } }
I20260810 21:09:43.061056 32652 leader_election.cc:302] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a3bf7053414d89bf79781ed7f0149d; no voters:
I20260810 21:09:43.061286 32652 leader_election.cc:288] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:43.061316 32659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:43.061434 32659 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [term 1 LEADER]: Becoming Leader. State: Replica: f9a3bf7053414d89bf79781ed7f0149d, State: Running, Role: LEADER
I20260810 21:09:43.061548 32652 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:43.061517 32659 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a3bf7053414d89bf79781ed7f0149d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35697 } }
I20260810 21:09:43.061867 32659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a3bf7053414d89bf79781ed7f0149d. Latest consensus state: current_term: 1 leader_uuid: "f9a3bf7053414d89bf79781ed7f0149d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a3bf7053414d89bf79781ed7f0149d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35697 } } }
I20260810 21:09:43.061967 32659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:43.062135 32660 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a3bf7053414d89bf79781ed7f0149d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a3bf7053414d89bf79781ed7f0149d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35697 } } }
I20260810 21:09:43.062209 32660 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:43.062350 32667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:43.063086 32667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:43.064868 32667 catalog_manager.cc:1347] Generated new cluster ID: 1c2660b347104038851cf89cda526779
I20260810 21:09:43.064934 32667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:43.069656 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.057610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.27.117.126:35697)
0810 21:09:43.057806 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:43.057821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:43.057847 (+ 26us) server_negotiation.cc:407] Connection header received
0810 21:09:43.060002 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:43.060009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:43.060091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:43.060230 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:43.060897 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.062200 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.063419 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.063952 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.064066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.066470 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:43.066474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:43.066474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:43.066486 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:43.068014 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.068397 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.068402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.068402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.068498 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.068752 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.068753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.068753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.069040 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:43.069203 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:43.069310 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 21:09:43.069452 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260810 21:09:43.076982 32667 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:43.077469 32667 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:43.082705 32667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a3bf7053414d89bf79781ed7f0149d: Generated new TSK 0
I20260810 21:09:43.082865 32667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:43.107313 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35697
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:43.187638 32700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:43.187870 32700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:43.187911 32700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:43.189124 32700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:43.189193 32700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:43.189239 32700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:43.190696 32700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:43.190975 32700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:43.191206 32700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:43.196048 32705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.196095 32716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.196105 32711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:43.196244 32700 server_base.cc:1034] running on GCE node
I20260810 21:09:43.196578 32700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:43.196893 32700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:43.198056 32700 hybrid_clock.cc:648] HybridClock initialized: now 1786396183198035 us; error 33 us; skew 500 ppm
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:43.200526 32700 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:43.200968 32700 webserver.cc:460] Webserver started at http://127.27.117.65:32967/ using document root <none> and password file <none>
I20260810 21:09:43.201167 32700 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:43.201210 32700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:43.201297 32700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:43.202255 32700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "7c201c95a55b4d9e8d4d7353d53af9d8"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "2fc583e017c33602eccad0cca3ddef77"
server_key_iv: "5253bdf5d84fd6497df73f420c6a66cb"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.202558 32700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "7c201c95a55b4d9e8d4d7353d53af9d8"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "2fc583e017c33602eccad0cca3ddef77"
server_key_iv: "5253bdf5d84fd6497df73f420c6a66cb"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.203795 32700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:43.204455 32724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.204644 32700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:43.204733 32700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "7c201c95a55b4d9e8d4d7353d53af9d8"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "2fc583e017c33602eccad0cca3ddef77"
server_key_iv: "5253bdf5d84fd6497df73f420c6a66cb"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.204826 32700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:43.227806 32700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:43.230075 32700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:43.230182 32700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:43.230341 32700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:43.230655 32700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:43.230690 32700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.230711 32700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:43.230759 32700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.236706 32700 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:39037
I20260810 21:09:43.236800 433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:39037 every 8 connection(s)
I20260810 21:09:43.237109 32700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:43.242136 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 32700
I20260810 21:09:43.242211 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:43.242376 28117 external_mini_cluster.cc:1428] Setting key 05efa9ca3de91c28c6e0fae689f7c55d
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:43.246487 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.237847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35697 (local address 127.27.117.65:40565)
0810 21:09:43.238042 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:43.238094 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:43.238544 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:43.238760 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:43.238764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:43.239294 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:43.239476 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:43.239480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.240141 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.240142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:43.241022 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:43.241025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.241366 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.241367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:43.241465 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.241878 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:43.241885 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:43.243552 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:43.245125 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:43.245132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:43.245134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:43.245440 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:43.245635 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:43.245636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:43.245636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:43.245786 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:43.246003 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:43.246008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:43.246148 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 21:09:43.246233 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 21:09:43.246340 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":120,"threads_started":1}
I20260810 21:09:43.246507 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.237891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:40565 (local address 127.27.117.126:35697)
0810 21:09:43.237989 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:43.237991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:43.238195 (+ 204us) server_negotiation.cc:407] Connection header received
0810 21:09:43.238626 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:43.238627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:43.238638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:43.238691 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:43.239556 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.240061 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.241099 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.241292 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.241346 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.243637 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:43.243638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:43.243639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:43.243645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:43.245046 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.245514 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.245515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.245515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.245570 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.245852 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.245853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.245853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.245945 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:43.245994 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:43.246288 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 21:09:43.246376 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 21:09:43.247176 434 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35697
I20260810 21:09:43.247282 434 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:43.247468 434 heartbeater.cc:511] Master 127.27.117.126:35697 requested a full tablet report, sending...
I20260810 21:09:43.248062 32593 ts_manager.cc:194] Registered new tserver with Master: 7c201c95a55b4d9e8d4d7353d53af9d8 (127.27.117.65:39037)
I20260810 21:09:43.249073 32593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:40565
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:43.267978 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35697
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:43.347994 443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:43.348177 443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:43.348199 443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:43.349292 443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:43.349327 443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:43.349354 443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:43.350708 443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:43.350898 443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:43.351069 443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:43.355821 452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.355814 449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.355815 450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:43.356142 443 server_base.cc:1034] running on GCE node
I20260810 21:09:43.356283 443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:43.356457 443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:43.357604 443 hybrid_clock.cc:648] HybridClock initialized: now 1786396183357580 us; error 39 us; skew 500 ppm
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:43.360044 443 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:43.360491 443 webserver.cc:460] Webserver started at http://127.27.117.66:45757/ using document root <none> and password file <none>
I20260810 21:09:43.360708 443 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:43.360752 443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:43.360955 443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:43.362007 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "4dea10f3900d43998775589d8b9704b4"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "b26814203f76d9f832b59b26a293c1dd"
server_key_iv: "729cd521a2564316f8973e1ee004794c"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.362402 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "4dea10f3900d43998775589d8b9704b4"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "b26814203f76d9f832b59b26a293c1dd"
server_key_iv: "729cd521a2564316f8973e1ee004794c"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.363710 443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:43.364389 463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.364538 443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:43.364655 443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "4dea10f3900d43998775589d8b9704b4"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "b26814203f76d9f832b59b26a293c1dd"
server_key_iv: "729cd521a2564316f8973e1ee004794c"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.364723 443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:43.375617 443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:43.377784 443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:43.377955 443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:43.378159 443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:43.378443 443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:43.378495 443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.378526 443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:43.378552 443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.384377 443 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36315
I20260810 21:09:43.384428 654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36315 every 8 connection(s)
I20260810 21:09:43.384706 443 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:43.392227 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 443
I20260810 21:09:43.392307 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:43.392520 28117 external_mini_cluster.cc:1428] Setting key 98423e0a155cf3d2189fb10c88b9ebf7
I20260810 21:09:43.394228 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.385447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:59271 (local address 127.27.117.126:35697)
0810 21:09:43.385537 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:43.385540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:43.385734 (+ 194us) server_negotiation.cc:407] Connection header received
0810 21:09:43.386215 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:43.386216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:43.386245 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:43.386301 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:43.387178 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.387686 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.388657 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.388926 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.389010 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.391173 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:43.391174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:43.391175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:43.391181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:43.392753 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.393231 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.393233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.393233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.393296 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.393593 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.393594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.393594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.393691 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:43.393741 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:43.394093 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 21:09:43.394166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 21:09:43.394475 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.385363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35697 (local address 127.27.117.66:59271)
0810 21:09:43.385567 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:43.385631 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:43.386108 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:43.386363 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:43.386367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:43.386905 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:43.387077 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:43.387081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.387783 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.387784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:43.388548 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:43.388549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.389022 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.389023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:43.389140 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.389608 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:43.389616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:43.391081 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:43.392853 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:43.392862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:43.392863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:43.393159 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:43.393372 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:43.393373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:43.393373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:43.393521 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:43.393756 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:43.393762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:43.393922 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 21:09:43.394047 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 21:09:43.394221 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260810 21:09:43.395001 655 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35697
I20260810 21:09:43.395112 655 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:43.395314 655 heartbeater.cc:511] Master 127.27.117.126:35697 requested a full tablet report, sending...
I20260810 21:09:43.395653 32593 ts_manager.cc:194] Registered new tserver with Master: 4dea10f3900d43998775589d8b9704b4 (127.27.117.66:36315)
I20260810 21:09:43.396154 32593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:59271
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:43.418833 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35697
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:43.499485 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:43.499663 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:43.499686 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:43.500801 666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:43.500846 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:43.500871 666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:43.502120 666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:43.502274 666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:43.502417 666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:43.507115 680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.507174 682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:43.507156 677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:43.507421 666 server_base.cc:1034] running on GCE node
I20260810 21:09:43.507674 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:43.507923 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:43.509089 666 hybrid_clock.cc:648] HybridClock initialized: now 1786396183509074 us; error 33 us; skew 500 ppm
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:43.511593 666 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:43.512002 666 webserver.cc:460] Webserver started at http://127.27.117.67:40863/ using document root <none> and password file <none>
I20260810 21:09:43.512233 666 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:43.512279 666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:43.512431 666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:43.513495 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "e9abf63169a9442c8a4d8f14a1177e9f"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "6ceed578b790dfba9c7a39b1d641f937"
server_key_iv: "6cd5753d1f9ed99499b620eb8bbb3c31"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.513861 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "e9abf63169a9442c8a4d8f14a1177e9f"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "6ceed578b790dfba9c7a39b1d641f937"
server_key_iv: "6cd5753d1f9ed99499b620eb8bbb3c31"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.515161 666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:43.515887 694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.516083 666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:43.516178 666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "e9abf63169a9442c8a4d8f14a1177e9f"
format_stamp: "Formatted at 2026-08-10 21:09:43 on dist-test-slave-2kcd"
server_key: "6ceed578b790dfba9c7a39b1d641f937"
server_key_iv: "6cd5753d1f9ed99499b620eb8bbb3c31"
server_key_version: "encryptionkey@0"
I20260810 21:09:43.516251 666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:43.532263 666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:43.534185 666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:43.534389 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:43.534622 666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:43.534933 666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:43.534967 666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.534988 666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:43.535032 666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:43.541074 666 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:33315
I20260810 21:09:43.541117 887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:33315 every 8 connection(s)
I20260810 21:09:43.541421 666 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:43.543505 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 666
I20260810 21:09:43.543620 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:43.543833 28117 external_mini_cluster.cc:1428] Setting key 46c4ff529dbaf590b650139bfc6bd31d
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396183, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:43.551405 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.542138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:56993 (local address 127.27.117.126:35697)
0810 21:09:43.542196 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:43.542198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:43.542413 (+ 215us) server_negotiation.cc:407] Connection header received
0810 21:09:43.542982 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:43.542983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:43.542995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:43.543044 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:43.544339 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.545016 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.546005 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.546215 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.546321 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.548446 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:43.548448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:43.548448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:43.548457 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:43.549910 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.550380 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.550381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.550381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.550488 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:43.550759 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:43.550760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:43.550760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:43.550878 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:43.550948 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:43.551225 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 21:09:43.551305 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 21:09:43.551610 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:43.542104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35697 (local address 127.27.117.67:56993)
0810 21:09:43.542294 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:43.542329 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:43.542852 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:43.543128 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:43.543134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:43.543989 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:43.544248 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:43.544254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.545111 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.545112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:43.545892 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:43.545894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:43.546334 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:43.546335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:43.546470 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:43.547001 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:43.547010 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:43.548348 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:43.550015 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:43.550023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:43.550025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:43.550272 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:43.550572 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:43.550572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:43.550573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:43.550682 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:43.550952 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:43.550959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:43.551092 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 21:09:43.551201 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 21:09:43.551354 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260810 21:09:43.552073 890 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35697
I20260810 21:09:43.552155 890 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:43.552383 890 heartbeater.cc:511] Master 127.27.117.126:35697 requested a full tablet report, sending...
I20260810 21:09:43.552802 32593 ts_manager.cc:194] Registered new tserver with Master: e9abf63169a9442c8a4d8f14a1177e9f (127.27.117.67:33315)
I20260810 21:09:43.553203 32593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:56993
I20260810 21:09:43.555599 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 21:09:43.568158 32658 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:43.567005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.27.117.126:35697)
0810 21:09:43.567112 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:43.567114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:43.567242 (+ 128us) server_negotiation.cc:407] Connection header received
0810 21:09:43.567333 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:43.567335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:43.567354 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:43.567433 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:43.568068 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35594: BlockingRecv error: recv got EOF from 127.0.0.1:35594 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 21:09:43.569234 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 32700
I20260810 21:09:43.579337 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 443
I20260810 21:09:43.592430 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 666
I20260810 21:09:43.608019 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 32541
2026-08-10T21:09:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2110 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[908](info): set up 2 sockets
Aug 10 21:09:43 dist-test-slave-2kcd krb5kdc[908](info): commencing operation
krb5kdc: starting...
W20260810 21:09:44.883179 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:44 dist-test-slave-2kcd krb5kdc[908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:45.001726 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45917
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:45.084403 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:45.084641 933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:45.084681 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:45.085826 933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:45.085901 933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:45.085927 933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:45.085956 933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:45.085987 933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:45.087467 933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:45917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:45.087736 933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:45.087966 933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:45.092732 945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:45.092756 944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:45.092864 933 server_base.cc:1034] running on GCE node
W20260810 21:09:45.092741 947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:45.093142 933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:45.093430 933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:45.094585 933 hybrid_clock.cc:648] HybridClock initialized: now 1786396185094574 us; error 32 us; skew 500 ppm
Aug 10 21:09:45 dist-test-slave-2kcd krb5kdc[908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396185, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:45.097141 933 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:45.097622 933 webserver.cc:460] Webserver started at http://127.27.117.126:41307/ using document root <none> and password file <none>
I20260810 21:09:45.097844 933 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:45.097896 933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:45.098066 933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:45.099099 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "484edb92c612436c90b660bee001fc2c"
format_stamp: "Formatted at 2026-08-10 21:09:45 on dist-test-slave-2kcd"
server_key: "dc7e59dad07de6edfed0815e08f4ec14"
server_key_iv: "9bc39e84ad55468b602112b67a602b8f"
server_key_version: "encryptionkey@0"
I20260810 21:09:45.099457 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "484edb92c612436c90b660bee001fc2c"
format_stamp: "Formatted at 2026-08-10 21:09:45 on dist-test-slave-2kcd"
server_key: "dc7e59dad07de6edfed0815e08f4ec14"
server_key_iv: "9bc39e84ad55468b602112b67a602b8f"
server_key_version: "encryptionkey@0"
I20260810 21:09:45.100837 933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:09:45.101614 959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:45.101801 933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:45.101918 933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "484edb92c612436c90b660bee001fc2c"
format_stamp: "Formatted at 2026-08-10 21:09:45 on dist-test-slave-2kcd"
server_key: "dc7e59dad07de6edfed0815e08f4ec14"
server_key_iv: "9bc39e84ad55468b602112b67a602b8f"
server_key_version: "encryptionkey@0"
I20260810 21:09:45.101994 933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:45.115485 933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:45.117928 933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:45.118099 933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:45.122328 933 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:45917
I20260810 21:09:45.122360 1052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:45917 every 8 connection(s)
I20260810 21:09:45.122992 933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:45.123653 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:45.126067 1053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c: Bootstrap starting.
I20260810 21:09:45.126236 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 933
I20260810 21:09:45.126341 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:45.126535 28117 external_mini_cluster.cc:1428] Setting key f65473f0fa57ccc7d4faab7422dec63e
I20260810 21:09:45.126852 1053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:45.127230 1053 log.cc:824] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:45.128005 1053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c: No bootstrap required, opened a new log
I20260810 21:09:45.130033 1053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484edb92c612436c90b660bee001fc2c" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45917 } }
I20260810 21:09:45.130191 1053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:45.130240 1053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484edb92c612436c90b660bee001fc2c, State: Initialized, Role: FOLLOWER
I20260810 21:09:45.130390 1053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484edb92c612436c90b660bee001fc2c" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45917 } }
I20260810 21:09:45.130470 1053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:45.130514 1053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:45.130613 1053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:45.131268 1053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484edb92c612436c90b660bee001fc2c" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45917 } }
I20260810 21:09:45.131374 1053 leader_election.cc:302] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484edb92c612436c90b660bee001fc2c; no voters:
I20260810 21:09:45.131527 1053 leader_election.cc:288] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:45.131583 1062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:45.131808 1062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [term 1 LEADER]: Becoming Leader. State: Replica: 484edb92c612436c90b660bee001fc2c, State: Running, Role: LEADER
I20260810 21:09:45.131839 1053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:45.132025 1062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484edb92c612436c90b660bee001fc2c" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45917 } }
I20260810 21:09:45.132895 1064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484edb92c612436c90b660bee001fc2c. Latest consensus state: current_term: 1 leader_uuid: "484edb92c612436c90b660bee001fc2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484edb92c612436c90b660bee001fc2c" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45917 } } }
I20260810 21:09:45.132925 1065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484edb92c612436c90b660bee001fc2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484edb92c612436c90b660bee001fc2c" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45917 } } }
I20260810 21:09:45.133023 1065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:45.132972 1064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c [sys.catalog]: This master's current role is: LEADER
Aug 10 21:09:45 dist-test-slave-2kcd krb5kdc[908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
W20260810 21:09:45.133946 1086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:09:45.134040 1086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:09:45.134135 1076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:45.134352 1076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:45.135944 1076 catalog_manager.cc:1347] Generated new cluster ID: d3ab33625d464818b02ee1dbfa893f78
I20260810 21:09:45.136066 1076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:45.138003 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:45.127162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.27.117.126:45917)
0810 21:09:45.127352 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:45.127362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:45.127394 (+ 32us) server_negotiation.cc:407] Connection header received
0810 21:09:45.127937 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:45.127945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:45.127994 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:45.128149 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:45.128811 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:45.129672 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:45.130586 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:45.131216 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:45.131339 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:45.134207 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:45.134211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:45.134211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:45.134223 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:45.136379 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:45.136725 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:45.136728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:45.136728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:45.136867 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:45.137090 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:45.137091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:45.137091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:45.137322 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:45.137509 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:45.137586 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 21:09:45.137724 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260810 21:09:45.139156 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:09:45.145924 1076 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:45.146669 1076 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:45.152138 1076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484edb92c612436c90b660bee001fc2c: Generated new TSK 0
I20260810 21:09:45.152308 1076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:09:45.155552 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:45.151819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.27.117.126:45917)
0810 21:09:45.151854 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:45.151857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:45.151953 (+ 96us) server_negotiation.cc:407] Connection header received
0810 21:09:45.152044 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:45.152045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:45.152061 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:45.152143 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:45.152630 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:45.153181 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:45.154238 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:45.154432 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:45.154482 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:45.155142 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:45.155143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:45.155143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:45.155309 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:45.155324 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:45.155329 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:45.155378 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:45.155460 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 21:09:45.155482 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:09:45.158015 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 933
2026-08-10T21:09:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1543 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:45 dist-test-slave-2kcd krb5kdc[1105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:45 dist-test-slave-2kcd krb5kdc[1105](info): set up 2 sockets
Aug 10 21:09:45 dist-test-slave-2kcd krb5kdc[1105](info): commencing operation
krb5kdc: starting...
W20260810 21:09:46.437322 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:46 dist-test-slave-2kcd krb5kdc[1105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:46.555080 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42671
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:42671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:46.634177 1129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:46.634390 1129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:46.634426 1129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:46.635491 1129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:46.635552 1129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:46.635574 1129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:46.635593 1129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:46.635624 1129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:46.637146 1129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:42671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:46.637413 1129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:46.637636 1129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:46.642508 1139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:46.642554 1129 server_base.cc:1034] running on GCE node
W20260810 21:09:46.642485 1138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:46.642516 1141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:46.642908 1129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:46.643198 1129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:46.644364 1129 hybrid_clock.cc:648] HybridClock initialized: now 1786396186644351 us; error 45 us; skew 500 ppm
Aug 10 21:09:46 dist-test-slave-2kcd krb5kdc[1105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:46.650916 1129 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:46.651393 1129 webserver.cc:460] Webserver started at http://127.27.117.126:37367/ using document root <none> and password file <none>
I20260810 21:09:46.651625 1129 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:46.651691 1129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:46.651861 1129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:46.652976 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a495d45d1b3d4c9c8af7dfaa847eb313"
format_stamp: "Formatted at 2026-08-10 21:09:46 on dist-test-slave-2kcd"
server_key: "a82c3c3fb32cbd62dcb3059f38310a1f"
server_key_iv: "ce25fcd5437db6722465a83456895001"
server_key_version: "encryptionkey@0"
I20260810 21:09:46.653332 1129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a495d45d1b3d4c9c8af7dfaa847eb313"
format_stamp: "Formatted at 2026-08-10 21:09:46 on dist-test-slave-2kcd"
server_key: "a82c3c3fb32cbd62dcb3059f38310a1f"
server_key_iv: "ce25fcd5437db6722465a83456895001"
server_key_version: "encryptionkey@0"
I20260810 21:09:46.654614 1129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:46.655288 1153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:46.655465 1129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:46.655570 1129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a495d45d1b3d4c9c8af7dfaa847eb313"
format_stamp: "Formatted at 2026-08-10 21:09:46 on dist-test-slave-2kcd"
server_key: "a82c3c3fb32cbd62dcb3059f38310a1f"
server_key_iv: "ce25fcd5437db6722465a83456895001"
server_key_version: "encryptionkey@0"
I20260810 21:09:46.655639 1129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:46.694475 1129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:46.696477 1129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:46.696656 1129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:46.700908 1129 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:42671
I20260810 21:09:46.700973 1248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:42671 every 8 connection(s)
I20260810 21:09:46.701294 1129 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:46.702412 1251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:46.704712 1251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313: Bootstrap starting.
I20260810 21:09:46.705341 1251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:46.705639 1251 log.cc:824] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:46.706354 1251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313: No bootstrap required, opened a new log
I20260810 21:09:46.707904 1251 raft_consensus.cc:348] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42671 } }
I20260810 21:09:46.708046 1251 raft_consensus.cc:374] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:46.708082 1251 raft_consensus.cc:729] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a495d45d1b3d4c9c8af7dfaa847eb313, State: Initialized, Role: FOLLOWER
I20260810 21:09:46.708184 1251 consensus_queue.cc:260] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42671 } }
I20260810 21:09:46.708262 1251 raft_consensus.cc:388] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:46.708294 1251 raft_consensus.cc:482] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:46.708353 1251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:46.709252 1251 raft_consensus.cc:504] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42671 } }
I20260810 21:09:46.709378 1251 leader_election.cc:302] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a495d45d1b3d4c9c8af7dfaa847eb313; no voters:
I20260810 21:09:46.709558 1251 leader_election.cc:288] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:46.709604 1257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:46.709780 1257 raft_consensus.cc:686] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [term 1 LEADER]: Becoming Leader. State: Replica: a495d45d1b3d4c9c8af7dfaa847eb313, State: Running, Role: LEADER
I20260810 21:09:46.709837 1251 sys_catalog.cc:564] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:46.710143 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1129
I20260810 21:09:46.710259 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:46.710434 28117 external_mini_cluster.cc:1428] Setting key 8206161599069748f6992fb5121b2035
I20260810 21:09:46.709932 1257 consensus_queue.cc:237] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42671 } }
I20260810 21:09:46.711174 1259 sys_catalog.cc:455] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a495d45d1b3d4c9c8af7dfaa847eb313. Latest consensus state: current_term: 1 leader_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42671 } } }
I20260810 21:09:46.711272 1259 sys_catalog.cc:458] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:46.711508 1258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a495d45d1b3d4c9c8af7dfaa847eb313" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42671 } } }
I20260810 21:09:46.711589 1258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:46.711714 1275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:46.712178 1275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:46.716826 1275 catalog_manager.cc:1347] Generated new cluster ID: 13be8bc090c849e68b4c41f8a8e5a101
I20260810 21:09:46.716884 1275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:46 dist-test-slave-2kcd krb5kdc[1105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:46.722038 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:46.710959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.27.117.126:42671)
0810 21:09:46.712927 (+ 1968us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:46.712937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:46.712971 (+ 34us) server_negotiation.cc:407] Connection header received
0810 21:09:46.713095 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:46.713101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:46.713168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:46.713314 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:46.713929 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:46.714849 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:46.715635 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:46.716054 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:46.716186 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:46.718631 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:46.718635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:46.718635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:46.718645 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:46.720379 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:46.720769 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:46.720772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:46.720772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:46.720922 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:46.721135 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:46.721136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:46.721136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:46.721389 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:46.721589 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:46.721664 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 21:09:46.721813 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1915,"thread_start_us":77,"threads_started":1}
I20260810 21:09:46.729763 1275 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:46.730257 1275 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:46.738471 1275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a495d45d1b3d4c9c8af7dfaa847eb313: Generated new TSK 0
I20260810 21:09:46.738654 1275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:46.773365 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42671
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:46.855654 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:46.855878 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:46.855918 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:46.857137 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:46.857198 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:46.857242 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:46.858835 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:46.859083 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:46.859352 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:46.864089 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:46.864188 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:46.864277 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:46.864518 1292 server_base.cc:1034] running on GCE node
I20260810 21:09:46.864715 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:46.865006 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:46.866182 1292 hybrid_clock.cc:648] HybridClock initialized: now 1786396186866172 us; error 39 us; skew 500 ppm
Aug 10 21:09:46 dist-test-slave-2kcd krb5kdc[1105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:46.868614 1292 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:46.869130 1292 webserver.cc:460] Webserver started at http://127.27.117.65:39415/ using document root <none> and password file <none>
I20260810 21:09:46.869361 1292 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:46.869429 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:46.869575 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:46.870543 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "b2469275b8674d6da1ffe49fa8d75d04"
format_stamp: "Formatted at 2026-08-10 21:09:46 on dist-test-slave-2kcd"
server_key: "ceed88f53159001ff29896f73fa39e80"
server_key_iv: "a53f3c157a59268e9bc03aa0ad041a45"
server_key_version: "encryptionkey@0"
I20260810 21:09:46.870906 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "b2469275b8674d6da1ffe49fa8d75d04"
format_stamp: "Formatted at 2026-08-10 21:09:46 on dist-test-slave-2kcd"
server_key: "ceed88f53159001ff29896f73fa39e80"
server_key_iv: "a53f3c157a59268e9bc03aa0ad041a45"
server_key_version: "encryptionkey@0"
I20260810 21:09:46.872202 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:46.872963 1311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:46.873136 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:46.873237 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "b2469275b8674d6da1ffe49fa8d75d04"
format_stamp: "Formatted at 2026-08-10 21:09:46 on dist-test-slave-2kcd"
server_key: "ceed88f53159001ff29896f73fa39e80"
server_key_iv: "a53f3c157a59268e9bc03aa0ad041a45"
server_key_version: "encryptionkey@0"
I20260810 21:09:46.873322 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:46.883260 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:46.884989 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:46.885138 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:46.885350 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:46.885675 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:46.885728 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:46.885768 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:46.885797 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:46.891800 1292 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:43803
I20260810 21:09:46.891855 1518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:43803 every 8 connection(s)
I20260810 21:09:46.892146 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:09:46 dist-test-slave-2kcd krb5kdc[1105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396186, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:46.898725 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1292
I20260810 21:09:46.898836 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:46.899340 28117 external_mini_cluster.cc:1428] Setting key e4c7a2df1b732a35d8b2bcdd1589b4aa
I20260810 21:09:46.901717 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:46.892769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42671 (local address 127.27.117.65:42427)
0810 21:09:46.892992 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:46.893036 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:46.893527 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:46.893777 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:46.893781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:46.894338 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:46.894520 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:46.894524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:46.895249 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:46.895251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:46.896036 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:46.896038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:46.896463 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:46.896464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:46.896594 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:46.897101 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:46.897109 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:46.898469 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:46.900446 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:46.900452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:46.900454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:46.900660 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:46.900890 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:46.900890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:46.900891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:46.900988 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:46.901265 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:46.901270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:46.901382 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 21:09:46.901460 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 21:09:46.901570 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260810 21:09:46.901749 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:46.892886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:42427 (local address 127.27.117.126:42671)
0810 21:09:46.892966 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:46.892968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:46.893123 (+ 155us) server_negotiation.cc:407] Connection header received
0810 21:09:46.893635 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:46.893636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:46.893649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:46.893702 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:46.894625 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:46.895149 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:46.896132 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:46.896374 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:46.896455 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:46.898587 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:46.898589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:46.898590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:46.898599 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:46.900373 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:46.900732 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:46.900733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:46.900734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:46.900830 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:46.901053 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:46.901054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:46.901054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:46.901201 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:46.901272 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:46.901506 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 21:09:46.901607 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 21:09:46.902496 1519 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42671
I20260810 21:09:46.902590 1519 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:46.902779 1519 heartbeater.cc:511] Master 127.27.117.126:42671 requested a full tablet report, sending...
I20260810 21:09:46.903352 1184 ts_manager.cc:194] Registered new tserver with Master: b2469275b8674d6da1ffe49fa8d75d04 (127.27.117.65:43803)
I20260810 21:09:46.903959 1184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:42427
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:46.924968 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42671
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:47.006213 1527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:47.006448 1527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:47.006486 1527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:47.007599 1527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:47.007666 1527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:47.007706 1527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:47.009199 1527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:47.009469 1527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:47.009703 1527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:47.014986 1542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:47.015012 1537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:47.015069 1527 server_base.cc:1034] running on GCE node
W20260810 21:09:47.015023 1540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:47.015401 1527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:47.015650 1527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:47.016831 1527 hybrid_clock.cc:648] HybridClock initialized: now 1786396187016809 us; error 38 us; skew 500 ppm
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[1105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396187, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:47.019352 1527 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:47.019784 1527 webserver.cc:460] Webserver started at http://127.27.117.66:35691/ using document root <none> and password file <none>
I20260810 21:09:47.020043 1527 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:47.020090 1527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:47.020241 1527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:47.021373 1527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "75713fdff5a0444f8f5be365767fb2c4"
format_stamp: "Formatted at 2026-08-10 21:09:47 on dist-test-slave-2kcd"
server_key: "660027ea782fed041837ce49ec5996aa"
server_key_iv: "9f4c8ad3e8dcdbe03d0d48828a1440d5"
server_key_version: "encryptionkey@0"
I20260810 21:09:47.021746 1527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "75713fdff5a0444f8f5be365767fb2c4"
format_stamp: "Formatted at 2026-08-10 21:09:47 on dist-test-slave-2kcd"
server_key: "660027ea782fed041837ce49ec5996aa"
server_key_iv: "9f4c8ad3e8dcdbe03d0d48828a1440d5"
server_key_version: "encryptionkey@0"
I20260810 21:09:47.023089 1527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:47.023801 1563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:47.023989 1527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:47.024087 1527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "75713fdff5a0444f8f5be365767fb2c4"
format_stamp: "Formatted at 2026-08-10 21:09:47 on dist-test-slave-2kcd"
server_key: "660027ea782fed041837ce49ec5996aa"
server_key_iv: "9f4c8ad3e8dcdbe03d0d48828a1440d5"
server_key_version: "encryptionkey@0"
I20260810 21:09:47.024170 1527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:47.037395 1527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:47.039740 1527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:47.039901 1527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:47.040135 1527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:47.040462 1527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:47.040519 1527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:47.040570 1527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:47.040599 1527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:47.047055 1527 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:40747
I20260810 21:09:47.047135 1771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:40747 every 8 connection(s)
I20260810 21:09:47.047461 1527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:47.049594 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1527
I20260810 21:09:47.049693 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:47.049904 28117 external_mini_cluster.cc:1428] Setting key 4c2a0dc05205c72e321de463c673bc80
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[1105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396187, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:47.058619 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:47.048257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:44987 (local address 127.27.117.126:42671)
0810 21:09:47.048324 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:47.048326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:47.048510 (+ 184us) server_negotiation.cc:407] Connection header received
0810 21:09:47.049148 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:47.049149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:47.049160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:47.049229 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:47.050279 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.050952 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.052557 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.052912 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.053008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:47.055121 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:47.055123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:47.055123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:47.055133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:47.057008 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:47.057400 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:47.057402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:47.057403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:47.057479 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:47.057852 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:47.057853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:47.057854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:47.058029 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:47.058117 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:47.058383 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 21:09:47.058487 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:09:47.058688 1779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:47.048215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42671 (local address 127.27.117.66:44987)
0810 21:09:47.048386 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:47.048428 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:47.049031 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:47.049283 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:47.049286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:47.049920 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:47.050165 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:47.050170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.051049 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.051050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:47.052393 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:47.052414 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.053011 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.053012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:47.053125 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:47.053663 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:47.053671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:47.055015 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:47.057096 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:47.057102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:47.057104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:47.057317 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:47.057617 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:47.057618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:47.057619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:47.057767 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:47.058103 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:47.058109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:47.058225 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 21:09:47.058314 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 21:09:47.058492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":100,"threads_started":1}
I20260810 21:09:47.059202 1776 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42671
I20260810 21:09:47.059331 1776 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:47.059541 1776 heartbeater.cc:511] Master 127.27.117.126:42671 requested a full tablet report, sending...
I20260810 21:09:47.059975 1184 ts_manager.cc:194] Registered new tserver with Master: 75713fdff5a0444f8f5be365767fb2c4 (127.27.117.66:40747)
I20260810 21:09:47.060350 1184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:44987
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:47.077817 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42671
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:47.160025 1782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:47.160211 1782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:47.160234 1782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:47.161383 1782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:47.161425 1782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:47.161451 1782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:47.162878 1782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:47.163049 1782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:47.163213 1782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:47.168077 1795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:47.168077 1794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:47.168092 1798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:47.168354 1782 server_base.cc:1034] running on GCE node
I20260810 21:09:47.168632 1782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:47.168907 1782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:47.170045 1782 hybrid_clock.cc:648] HybridClock initialized: now 1786396187170035 us; error 26 us; skew 500 ppm
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[1105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396187, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:47.172602 1782 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:47.173116 1782 webserver.cc:460] Webserver started at http://127.27.117.67:45343/ using document root <none> and password file <none>
I20260810 21:09:47.173332 1782 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:47.173377 1782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:47.173511 1782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:47.174499 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "7381ad86c0c94f7abc754415105a5e9a"
format_stamp: "Formatted at 2026-08-10 21:09:47 on dist-test-slave-2kcd"
server_key: "829b5ffc5133879d4db4012a910c1ba0"
server_key_iv: "c09f1fc3596c3d1ffb5bb6c7db25854b"
server_key_version: "encryptionkey@0"
I20260810 21:09:47.175032 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "7381ad86c0c94f7abc754415105a5e9a"
format_stamp: "Formatted at 2026-08-10 21:09:47 on dist-test-slave-2kcd"
server_key: "829b5ffc5133879d4db4012a910c1ba0"
server_key_iv: "c09f1fc3596c3d1ffb5bb6c7db25854b"
server_key_version: "encryptionkey@0"
I20260810 21:09:47.176436 1782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:09:47.177280 1806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:47.177479 1782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:09:47.177561 1782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "7381ad86c0c94f7abc754415105a5e9a"
format_stamp: "Formatted at 2026-08-10 21:09:47 on dist-test-slave-2kcd"
server_key: "829b5ffc5133879d4db4012a910c1ba0"
server_key_iv: "c09f1fc3596c3d1ffb5bb6c7db25854b"
server_key_version: "encryptionkey@0"
I20260810 21:09:47.177614 1782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:47.187801 1782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:47.189618 1782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:47.189725 1782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:47.189936 1782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:47.190181 1782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:47.190212 1782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:47.190232 1782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:47.190245 1782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:47.195923 1782 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:42023
I20260810 21:09:47.196002 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:42023 every 8 connection(s)
I20260810 21:09:47.196288 1782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[1105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396187, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:47.202863 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1782
I20260810 21:09:47.202978 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:47.203171 28117 external_mini_cluster.cc:1428] Setting key a8b175d67b19adb7679e2b00bb26318a
I20260810 21:09:47.205585 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:47.197086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:37541 (local address 127.27.117.126:42671)
0810 21:09:47.197172 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:47.197174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:47.197349 (+ 175us) server_negotiation.cc:407] Connection header received
0810 21:09:47.197957 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:47.197959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:47.197971 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:47.198034 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:47.198963 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.199473 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.200461 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.200675 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.200754 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:47.202866 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:47.202872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:47.202872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:47.202879 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:47.204473 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:47.204890 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:47.204890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:47.204891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:47.204943 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:47.205125 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:47.205126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:47.205126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:47.205217 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:47.205263 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:47.205471 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 21:09:47.205526 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:09:47.205757 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:47.197044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42671 (local address 127.27.117.67:37541)
0810 21:09:47.197219 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:47.197264 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:47.197835 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:47.198099 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:47.198103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:47.198665 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:47.198841 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:47.198845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.199570 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.199571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:47.200360 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:47.200362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.200762 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.200763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:47.200952 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:47.201445 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:47.201453 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:47.202753 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:47.204543 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:47.204549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:47.204550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:47.204767 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:47.204987 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:47.204987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:47.204987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:47.205077 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:47.205270 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:47.205274 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:47.205376 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 21:09:47.205460 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 21:09:47.205569 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260810 21:09:47.206313 1993 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42671
I20260810 21:09:47.206393 1993 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:47.206596 1993 heartbeater.cc:511] Master 127.27.117.126:42671 requested a full tablet report, sending...
I20260810 21:09:47.206991 1184 ts_manager.cc:194] Registered new tserver with Master: 7381ad86c0c94f7abc754415105a5e9a (127.27.117.67:42023)
I20260810 21:09:47.207353 1184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:37541
I20260810 21:09:47.214977 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[1105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396187, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[1105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396187, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:47.237977 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:47.231057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.27.117.126:42671)
0810 21:09:47.231126 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:47.231129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:47.231225 (+ 96us) server_negotiation.cc:407] Connection header received
0810 21:09:47.231307 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:47.231308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:47.231323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:47.231395 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:47.231937 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.232417 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.233230 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:47.233418 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:47.233488 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:47.235579 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:47.235580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:47.235580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:47.235588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:47.236755 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:47.237180 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:47.237182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:47.237182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:47.237238 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:47.237439 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:47.237440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:47.237440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:47.237589 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:47.237697 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:47.237795 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 21:09:47.237873 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 21:09:47.238204 1184 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54546
I20260810 21:09:47.238618 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1292
I20260810 21:09:47.258543 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1527
I20260810 21:09:47.273615 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1782
I20260810 21:09:47.284881 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1129
2026-08-10T21:09:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2136 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[2009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[2009](info): set up 2 sockets
Aug 10 21:09:47 dist-test-slave-2kcd krb5kdc[2009](info): commencing operation
krb5kdc: starting...
W20260810 21:09:48.566862 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:48 dist-test-slave-2kcd krb5kdc[2009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:48.681725 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:34935
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:48.763589 2026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:48.763778 2026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:48.763800 2026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:48.764968 2026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:48.765010 2026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:48.765024 2026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:48.765038 2026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:48.765080 2026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:48.766500 2026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:48.766749 2026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:48.766966 2026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:48.771816 2036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:48.771853 2043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:48.771823 2038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:48.772030 2026 server_base.cc:1034] running on GCE node
I20260810 21:09:48.772282 2026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:48.772632 2026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:48.773792 2026 hybrid_clock.cc:648] HybridClock initialized: now 1786396188773779 us; error 27 us; skew 500 ppm
Aug 10 21:09:48 dist-test-slave-2kcd krb5kdc[2009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396188, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:48.776337 2026 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:48.776860 2026 webserver.cc:460] Webserver started at http://127.27.117.126:37809/ using document root <none> and password file <none>
I20260810 21:09:48.777087 2026 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:48.777163 2026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:48.777300 2026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:48.778352 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc"
format_stamp: "Formatted at 2026-08-10 21:09:48 on dist-test-slave-2kcd"
server_key: "c1f5dae47c0d88fc957460145c98595c"
server_key_iv: "aaaca933f20d6c81c9475480c4bb83bf"
server_key_version: "encryptionkey@0"
I20260810 21:09:48.778716 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc"
format_stamp: "Formatted at 2026-08-10 21:09:48 on dist-test-slave-2kcd"
server_key: "c1f5dae47c0d88fc957460145c98595c"
server_key_iv: "aaaca933f20d6c81c9475480c4bb83bf"
server_key_version: "encryptionkey@0"
I20260810 21:09:48.780014 2026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:48.780720 2052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:48.780928 2026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:48.781032 2026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc"
format_stamp: "Formatted at 2026-08-10 21:09:48 on dist-test-slave-2kcd"
server_key: "c1f5dae47c0d88fc957460145c98595c"
server_key_iv: "aaaca933f20d6c81c9475480c4bb83bf"
server_key_version: "encryptionkey@0"
I20260810 21:09:48.781138 2026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:48.792568 2026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:48.794414 2026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:48.794589 2026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:48.799078 2026 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:34935
I20260810 21:09:48.799134 2148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:34935 every 8 connection(s)
I20260810 21:09:48.799631 2026 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:48.800323 2152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:48.802783 2152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc: Bootstrap starting.
I20260810 21:09:48.803426 2152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:48.803738 2152 log.cc:824] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:48.804455 2152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc: No bootstrap required, opened a new log
I20260810 21:09:48.806015 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2026
I20260810 21:09:48.806138 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:48.806334 28117 external_mini_cluster.cc:1428] Setting key ebdff0ce5627a2d6bf5e4a3e76b27376
I20260810 21:09:48.806361 2152 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34935 } }
I20260810 21:09:48.806540 2152 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:48.806599 2152 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf16d6ff1f44d8c95e9f56cc5da9dcc, State: Initialized, Role: FOLLOWER
I20260810 21:09:48.806746 2152 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34935 } }
I20260810 21:09:48.806833 2152 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:48.806885 2152 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:48.806937 2152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:48.807765 2152 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34935 } }
I20260810 21:09:48.807950 2152 leader_election.cc:302] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf16d6ff1f44d8c95e9f56cc5da9dcc; no voters:
I20260810 21:09:48.808139 2152 leader_election.cc:288] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:48.808171 2159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:48.808290 2159 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [term 1 LEADER]: Becoming Leader. State: Replica: fbf16d6ff1f44d8c95e9f56cc5da9dcc, State: Running, Role: LEADER
I20260810 21:09:48.808393 2159 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34935 } }
I20260810 21:09:48.808415 2152 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:48.808746 2163 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf16d6ff1f44d8c95e9f56cc5da9dcc. Latest consensus state: current_term: 1 leader_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34935 } } }
I20260810 21:09:48.808867 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:48.809057 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf16d6ff1f44d8c95e9f56cc5da9dcc" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 34935 } } }
I20260810 21:09:48.809129 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc [sys.catalog]: This master's current role is: LEADER
W20260810 21:09:48.811424 2185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:09:48.811492 2185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:09:48.811583 2170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:48.811755 2170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:48.813318 2170 catalog_manager.cc:1347] Generated new cluster ID: ca42fea8c2b5449caffe085776a02954
I20260810 21:09:48.813375 2170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:48 dist-test-slave-2kcd krb5kdc[2009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:48.818756 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:48.806836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.27.117.126:34935)
0810 21:09:48.807632 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:48.807641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:48.807761 (+ 120us) server_negotiation.cc:407] Connection header received
0810 21:09:48.807921 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:48.807929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:48.807984 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:48.808148 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:48.809394 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:48.810581 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:48.811590 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:48.812191 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:48.812305 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:48.815145 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:48.815149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:48.815149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:48.815160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:48.817141 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:48.817545 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:48.817548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:48.817548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:48.817665 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:48.817885 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:48.817886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:48.817886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:48.818167 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:48.818316 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:48.818453 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 21:09:48.818573 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":702,"threads_started":1}
I20260810 21:09:48.830148 2170 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:48.830662 2170 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:48.836346 2170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf16d6ff1f44d8c95e9f56cc5da9dcc: Generated new TSK 0
I20260810 21:09:48.836544 2170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:48.865448 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34935
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:48.947902 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:48.948100 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:48.948127 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:48.949431 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:48.949477 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:48.949506 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:48.951104 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:48.951360 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:48.951611 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:48.956641 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:48.956761 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:48.956676 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:48.956986 2193 server_base.cc:1034] running on GCE node
I20260810 21:09:48.957254 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:48.957525 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:48.958698 2193 hybrid_clock.cc:648] HybridClock initialized: now 1786396188958685 us; error 30 us; skew 500 ppm
Aug 10 21:09:48 dist-test-slave-2kcd krb5kdc[2009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396188, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:48.961303 2193 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:48.961668 2193 webserver.cc:460] Webserver started at http://127.27.117.65:40493/ using document root <none> and password file <none>
I20260810 21:09:48.961856 2193 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:48.961895 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:48.961979 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:48.962893 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "771b2fc1a2e4469db75019ab58f52c0d"
format_stamp: "Formatted at 2026-08-10 21:09:48 on dist-test-slave-2kcd"
server_key: "c83c93d4b4b9f597a90dfcd4602eb376"
server_key_iv: "e12480b3a228ee7c0f476f2930bb6c59"
server_key_version: "encryptionkey@0"
I20260810 21:09:48.963192 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "771b2fc1a2e4469db75019ab58f52c0d"
format_stamp: "Formatted at 2026-08-10 21:09:48 on dist-test-slave-2kcd"
server_key: "c83c93d4b4b9f597a90dfcd4602eb376"
server_key_iv: "e12480b3a228ee7c0f476f2930bb6c59"
server_key_version: "encryptionkey@0"
I20260810 21:09:48.964449 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:09:48.965224 2217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:48.965404 2193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:48.965499 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "771b2fc1a2e4469db75019ab58f52c0d"
format_stamp: "Formatted at 2026-08-10 21:09:48 on dist-test-slave-2kcd"
server_key: "c83c93d4b4b9f597a90dfcd4602eb376"
server_key_iv: "e12480b3a228ee7c0f476f2930bb6c59"
server_key_version: "encryptionkey@0"
I20260810 21:09:48.965554 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:48.989133 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:48.991063 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:48.991176 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:48.991376 2193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:48.991642 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:48.991673 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:48.991695 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:48.991708 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:48.997584 2193 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:44901
I20260810 21:09:48.997671 2415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:44901 every 8 connection(s)
I20260810 21:09:48.997992 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:49.000509 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2193
I20260810 21:09:49.000592 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:49.000828 28117 external_mini_cluster.cc:1428] Setting key e216b9fe9e93dfbd8327d6fe4a04995c
Aug 10 21:09:48 dist-test-slave-2kcd krb5kdc[2009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396188, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:49.008474 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:48.998750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:35695 (local address 127.27.117.126:34935)
0810 21:09:48.998813 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:48.998815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:48.999049 (+ 234us) server_negotiation.cc:407] Connection header received
0810 21:09:48.999969 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:48.999971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:48.999985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:49.000054 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:49.000715 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.001462 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.002441 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.002756 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.002833 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.005240 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:49.005241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:49.005242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:49.005249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:49.007053 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.007447 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.007449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.007449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.007517 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.007752 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.007753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.007754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.007910 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:49.007988 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:49.008253 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 21:09:49.008359 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:09:49.008498 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:48.998729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:34935 (local address 127.27.117.65:35695)
0810 21:09:48.998911 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:48.998952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:48.999878 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:49.000111 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:49.000114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:49.000359 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:49.000584 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:49.000591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.001557 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.001558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:49.002342 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:49.002344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.002845 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.002846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:49.002945 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.003507 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:49.003518 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:49.005134 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:49.007142 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:49.007148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:49.007150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:49.007368 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:49.007588 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:49.007589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:49.007589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:49.007686 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:49.007978 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:49.007983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:49.008111 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 21:09:49.008219 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 21:09:49.008340 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260810 21:09:49.009133 2417 heartbeater.cc:347] Connected to a master server at 127.27.117.126:34935
I20260810 21:09:49.009248 2417 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:49.009449 2417 heartbeater.cc:511] Master 127.27.117.126:34935 requested a full tablet report, sending...
I20260810 21:09:49.010037 2076 ts_manager.cc:194] Registered new tserver with Master: 771b2fc1a2e4469db75019ab58f52c0d (127.27.117.65:44901)
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
I20260810 21:09:49.010728 2076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:35695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:49.027422 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34935
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:49.109647 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:49.109865 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:49.109901 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:49.111032 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:49.111097 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:49.111140 2425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:49.112629 2425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:49.112910 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:49.113147 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:49.118038 2434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:49.118057 2435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:49.118057 2438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:49.118234 2425 server_base.cc:1034] running on GCE node
I20260810 21:09:49.118433 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:49.118660 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:49.119807 2425 hybrid_clock.cc:648] HybridClock initialized: now 1786396189119793 us; error 30 us; skew 500 ppm
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:49.122319 2425 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:49.122749 2425 webserver.cc:460] Webserver started at http://127.27.117.66:39901/ using document root <none> and password file <none>
I20260810 21:09:49.122975 2425 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:49.123045 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:49.123202 2425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:49.124212 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "3267a2bc2b4f44e3b12a410622aff4fb"
format_stamp: "Formatted at 2026-08-10 21:09:49 on dist-test-slave-2kcd"
server_key: "824c54bea84ae7624a29d948cd4b423e"
server_key_iv: "3030e98ea3c517e6e7aa38c649919e83"
server_key_version: "encryptionkey@0"
I20260810 21:09:49.124586 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "3267a2bc2b4f44e3b12a410622aff4fb"
format_stamp: "Formatted at 2026-08-10 21:09:49 on dist-test-slave-2kcd"
server_key: "824c54bea84ae7624a29d948cd4b423e"
server_key_iv: "3030e98ea3c517e6e7aa38c649919e83"
server_key_version: "encryptionkey@0"
I20260810 21:09:49.125965 2425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:49.126683 2447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:49.126845 2425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:49.126956 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "3267a2bc2b4f44e3b12a410622aff4fb"
format_stamp: "Formatted at 2026-08-10 21:09:49 on dist-test-slave-2kcd"
server_key: "824c54bea84ae7624a29d948cd4b423e"
server_key_iv: "3030e98ea3c517e6e7aa38c649919e83"
server_key_version: "encryptionkey@0"
I20260810 21:09:49.127031 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:49.134330 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:49.136029 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:49.136188 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:49.136399 2425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:49.136700 2425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:49.136757 2425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:49.136790 2425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:49.136861 2425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:49.142608 2425 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:44943
I20260810 21:09:49.142678 2650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:44943 every 8 connection(s)
I20260810 21:09:49.142989 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:49.152088 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2425
I20260810 21:09:49.152205 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:49.152410 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:49.143777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:54049 (local address 127.27.117.126:34935)
0810 21:09:49.143840 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:49.143842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:49.144032 (+ 190us) server_negotiation.cc:407] Connection header received
0810 21:09:49.145074 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:49.145076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:49.145089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:49.145146 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:49.145783 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.146375 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.147351 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.147613 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.147698 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.149886 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:49.149887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:49.149887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:49.149894 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:49.151102 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.151463 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.151463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.151464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.151516 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.151732 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.151733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.151733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.151828 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:49.151908 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:49.152244 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 21:09:49.152336 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:09:49.152415 28117 external_mini_cluster.cc:1428] Setting key a8667e948260cd486003f362e7616814
I20260810 21:09:49.152608 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:49.143747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:34935 (local address 127.27.117.66:54049)
0810 21:09:49.143906 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:49.143948 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:49.144961 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:49.145212 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:49.145216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:49.145493 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:49.145675 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:49.145679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.146488 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.146489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:49.147269 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:49.147271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.147707 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.147708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:49.147847 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.148370 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:49.148378 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:49.149795 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:49.151176 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:49.151183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:49.151184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:49.151396 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:49.151579 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:49.151579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:49.151580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:49.151676 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:49.151883 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:49.151887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:49.152081 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 21:09:49.152210 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 21:09:49.152380 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260810 21:09:49.153149 2653 heartbeater.cc:347] Connected to a master server at 127.27.117.126:34935
I20260810 21:09:49.153257 2653 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:49.153466 2653 heartbeater.cc:511] Master 127.27.117.126:34935 requested a full tablet report, sending...
I20260810 21:09:49.153854 2076 ts_manager.cc:194] Registered new tserver with Master: 3267a2bc2b4f44e3b12a410622aff4fb (127.27.117.66:44943)
I20260810 21:09:49.154341 2076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:54049
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:49.177590 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34935
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:49.258406 2665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:49.258647 2665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:49.258687 2665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:49.259830 2665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:49.259898 2665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:49.259944 2665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:49.261552 2665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:34935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:49.261823 2665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:49.262061 2665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:49.267462 2682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:49.267499 2665 server_base.cc:1034] running on GCE node
W20260810 21:09:49.267508 2677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:49.267462 2679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:49.267880 2665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:49.268132 2665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:49.269294 2665 hybrid_clock.cc:648] HybridClock initialized: now 1786396189269279 us; error 46 us; skew 500 ppm
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:49.271823 2665 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:49.272298 2665 webserver.cc:460] Webserver started at http://127.27.117.67:35151/ using document root <none> and password file <none>
I20260810 21:09:49.272547 2665 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:49.272616 2665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:49.272765 2665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:49.273805 2665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "1c3d9950db6e4489858fadb5090c1df1"
format_stamp: "Formatted at 2026-08-10 21:09:49 on dist-test-slave-2kcd"
server_key: "44eb1bfde69e8534395565cca1f68de6"
server_key_iv: "8999273deaab6f0db06d9c55f178898b"
server_key_version: "encryptionkey@0"
I20260810 21:09:49.274170 2665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "1c3d9950db6e4489858fadb5090c1df1"
format_stamp: "Formatted at 2026-08-10 21:09:49 on dist-test-slave-2kcd"
server_key: "44eb1bfde69e8534395565cca1f68de6"
server_key_iv: "8999273deaab6f0db06d9c55f178898b"
server_key_version: "encryptionkey@0"
I20260810 21:09:49.275508 2665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:49.276271 2698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:49.276448 2665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:49.276553 2665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "1c3d9950db6e4489858fadb5090c1df1"
format_stamp: "Formatted at 2026-08-10 21:09:49 on dist-test-slave-2kcd"
server_key: "44eb1bfde69e8534395565cca1f68de6"
server_key_iv: "8999273deaab6f0db06d9c55f178898b"
server_key_version: "encryptionkey@0"
I20260810 21:09:49.276635 2665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:49.297596 2665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:49.299990 2665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:49.300166 2665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:49.300413 2665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:49.300755 2665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:49.300832 2665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:49.300881 2665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:49.300904 2665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:49.306984 2665 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:40073
I20260810 21:09:49.307112 2911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:40073 every 8 connection(s)
I20260810 21:09:49.307421 2665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:49.312314 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2665
I20260810 21:09:49.312443 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:49.312668 28117 external_mini_cluster.cc:1428] Setting key 6ec131d7ccb4af1e137f4fe68bdca7cc
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:49.316968 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:49.308243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:44783 (local address 127.27.117.126:34935)
0810 21:09:49.308307 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:49.308309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:49.308565 (+ 256us) server_negotiation.cc:407] Connection header received
0810 21:09:49.309702 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:49.309703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:49.309741 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:49.309792 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:49.310468 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.310970 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.311976 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.312193 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.312273 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.314656 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:49.314657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:49.314658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:49.314665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:49.315792 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.316151 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.316152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.316152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.316201 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.316408 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.316409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.316409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.316511 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:49.316562 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:49.316822 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 21:09:49.316902 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:09:49.317195 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:49.308230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:34935 (local address 127.27.117.67:44783)
0810 21:09:49.308407 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:49.308469 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:49.309570 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:49.309876 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:49.309880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:49.310178 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:49.310365 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:49.310369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.311066 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.311068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:49.311862 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:49.311864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.312281 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.312282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:49.312435 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.313050 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:49.313058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:49.314565 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:49.315861 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:49.315866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:49.315868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:49.316084 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:49.316259 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:49.316260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:49.316260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:49.316354 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:49.316569 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:49.316573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:49.316680 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 21:09:49.316761 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:09:49.316941 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260810 21:09:49.317626 2913 heartbeater.cc:347] Connected to a master server at 127.27.117.126:34935
I20260810 21:09:49.317725 2913 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:49.317900 2913 heartbeater.cc:511] Master 127.27.117.126:34935 requested a full tablet report, sending...
I20260810 21:09:49.318262 2076 ts_manager.cc:194] Registered new tserver with Master: 1c3d9950db6e4489858fadb5090c1df1 (127.27.117.67:40073)
I20260810 21:09:49.318636 2076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:44783
I20260810 21:09:49.324404 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:49.348235 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:49.340538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.27.117.65:44901)
0810 21:09:49.340735 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:49.340740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:49.340751 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:49.340811 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:49.340814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:49.340840 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:49.340910 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:49.341456 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.342170 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.342867 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.343180 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.343235 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.345402 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:49.345411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:49.345411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:49.345418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:49.347131 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.347434 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.347437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.347438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.347505 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.347657 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.347657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.347658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.347774 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:49.347879 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:49.348015 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 21:09:49.348116 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:49.356339 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:49.349607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.27.117.126:34935)
0810 21:09:49.349673 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:49.349676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:49.349686 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:49.349760 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:49.349762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:49.349773 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:49.349827 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:49.350278 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.350764 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.351385 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:49.351564 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:49.351614 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:49.353803 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:49.353804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:49.353804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:49.353811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:49.354938 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.355376 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.355377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.355378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.355452 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:49.355679 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:49.355680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:49.355681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:49.355889 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:49.355993 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:49.356169 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 21:09:49.356254 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 21:09:49.356575 2076 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59528
I20260810 21:09:49.357059 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2193
I20260810 21:09:49.371026 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2425
I20260810 21:09:49.394726 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2665
I20260810 21:09:49.412729 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2026
2026-08-10T21:09:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2128 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2933](info): set up 2 sockets
Aug 10 21:09:49 dist-test-slave-2kcd krb5kdc[2933](info): commencing operation
krb5kdc: starting...
W20260810 21:09:50.687362 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:50 dist-test-slave-2kcd krb5kdc[2933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:50.804049 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40085
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:40085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:50.887159 2950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:50.887408 2950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:50.887445 2950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:50.888586 2950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:50.888653 2950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:50.888679 2950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:50.888711 2950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:50.888754 2950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:50.890302 2950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:40085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:50.890566 2950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:50.890816 2950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:50.895509 2964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:50.895524 2960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:50.895524 2961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:50.895725 2950 server_base.cc:1034] running on GCE node
I20260810 21:09:50.896008 2950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:50.896297 2950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:50.897459 2950 hybrid_clock.cc:648] HybridClock initialized: now 1786396190897435 us; error 35 us; skew 500 ppm
Aug 10 21:09:50 dist-test-slave-2kcd krb5kdc[2933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396190, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:50.899936 2950 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:50.900296 2950 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:50.900345 2950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:50.900435 2950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:50.901505 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "2f433a469ad4404d8919763d058aaa14"
format_stamp: "Formatted at 2026-08-10 21:09:50 on dist-test-slave-2kcd"
server_key: "7f7758e05a593fdac078fcb404db1044"
server_key_iv: "1e1391894b17adcdac29c229f41f0246"
server_key_version: "encryptionkey@0"
I20260810 21:09:50.901790 2950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "2f433a469ad4404d8919763d058aaa14"
format_stamp: "Formatted at 2026-08-10 21:09:50 on dist-test-slave-2kcd"
server_key: "7f7758e05a593fdac078fcb404db1044"
server_key_iv: "1e1391894b17adcdac29c229f41f0246"
server_key_version: "encryptionkey@0"
I20260810 21:09:50.903021 2950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:50.903702 2980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:50.903888 2950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:50.903962 2950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "2f433a469ad4404d8919763d058aaa14"
format_stamp: "Formatted at 2026-08-10 21:09:50 on dist-test-slave-2kcd"
server_key: "7f7758e05a593fdac078fcb404db1044"
server_key_iv: "1e1391894b17adcdac29c229f41f0246"
server_key_version: "encryptionkey@0"
I20260810 21:09:50.904016 2950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:50.924077 2950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:50.925904 2950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:50.926016 2950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:50.930138 2950 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:40085
I20260810 21:09:50.930166 3079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:40085 every 8 connection(s)
I20260810 21:09:50.930455 2950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:50.931291 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2950
I20260810 21:09:50.931430 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:50.931648 28117 external_mini_cluster.cc:1428] Setting key 555d72ca707315f0ea52d69e2ef13a6e
I20260810 21:09:50.931816 3080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:50.935158 3080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14: Bootstrap starting.
I20260810 21:09:50.935889 3080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:50.936182 3080 log.cc:824] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:50.937031 3080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14: No bootstrap required, opened a new log
Aug 10 21:09:50 dist-test-slave-2kcd krb5kdc[2933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:50.939368 3080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f433a469ad4404d8919763d058aaa14" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40085 } }
I20260810 21:09:50.939546 3080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:50.939585 3080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f433a469ad4404d8919763d058aaa14, State: Initialized, Role: FOLLOWER
I20260810 21:09:50.939728 3080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f433a469ad4404d8919763d058aaa14" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40085 } }
I20260810 21:09:50.939819 3080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:50.939869 3080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:50.939947 3080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:50.940627 3080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f433a469ad4404d8919763d058aaa14" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40085 } }
I20260810 21:09:50.940737 3080 leader_election.cc:302] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f433a469ad4404d8919763d058aaa14; no voters:
I20260810 21:09:50.940924 3080 leader_election.cc:288] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:50.940975 3086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:50.941140 3086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [term 1 LEADER]: Becoming Leader. State: Replica: 2f433a469ad4404d8919763d058aaa14, State: Running, Role: LEADER
I20260810 21:09:50.941196 3080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:50.941228 3086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f433a469ad4404d8919763d058aaa14" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40085 } }
I20260810 21:09:50.941643 3087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f433a469ad4404d8919763d058aaa14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f433a469ad4404d8919763d058aaa14" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40085 } } }
I20260810 21:09:50.941753 3087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:50.941638 3088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f433a469ad4404d8919763d058aaa14. Latest consensus state: current_term: 1 leader_uuid: "2f433a469ad4404d8919763d058aaa14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f433a469ad4404d8919763d058aaa14" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40085 } } }
I20260810 21:09:50.941926 3088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:50.942040 3097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:50.942714 3097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:50.943280 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:50.932219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.27.117.126:40085)
0810 21:09:50.932423 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:50.932432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:50.932467 (+ 35us) server_negotiation.cc:407] Connection header received
0810 21:09:50.932999 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:50.933006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:50.933050 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:50.933160 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:50.934036 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:50.934911 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:50.935996 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:50.936461 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:50.936549 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:50.939081 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:50.939086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:50.939086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:50.939097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:50.941090 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:50.941591 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:50.941594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:50.941594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:50.941712 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:50.941968 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:50.941968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:50.941969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:50.942278 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:50.942886 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:50.942988 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 21:09:50.943139 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260810 21:09:50.945780 3097 catalog_manager.cc:1347] Generated new cluster ID: 0edc508fe2954fb09c4923d02422812e
I20260810 21:09:50.945838 3097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:50.983529 3097 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:50.984095 3097 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:50.987300 3097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14: Generated new TSK 0
I20260810 21:09:50.987480 3097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:51.023221 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40085
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:51.104838 3121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:51.105049 3121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:51.105075 3121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:51.106248 3121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:51.106287 3121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:51.106336 3121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:51.107978 3121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:51.108211 3121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:51.108446 3121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:51.113340 3136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.113332 3142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.113332 3139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:51.113580 3121 server_base.cc:1034] running on GCE node
I20260810 21:09:51.113837 3121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:51.114133 3121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:51.115298 3121 hybrid_clock.cc:648] HybridClock initialized: now 1786396191115294 us; error 37 us; skew 500 ppm
Aug 10 21:09:51 dist-test-slave-2kcd krb5kdc[2933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:51.117983 3121 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:51.118342 3121 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:51.118415 3121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:51.118574 3121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:51.119632 3121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "31565bd9c3b14a9cafa27c80952efbd7"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "36d7e626c9f193906c5bfaccfe653fc9"
server_key_iv: "42b21df8c195028d8cc6319c8954d7cd"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.120023 3121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "31565bd9c3b14a9cafa27c80952efbd7"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "36d7e626c9f193906c5bfaccfe653fc9"
server_key_iv: "42b21df8c195028d8cc6319c8954d7cd"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.121395 3121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:51.122148 3157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.122318 3121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:51.122412 3121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "31565bd9c3b14a9cafa27c80952efbd7"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "36d7e626c9f193906c5bfaccfe653fc9"
server_key_iv: "42b21df8c195028d8cc6319c8954d7cd"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.122495 3121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:51.131526 3121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:51.133924 3121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:51.134102 3121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:51.134342 3121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:51.134665 3121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:51.134722 3121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.134763 3121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:51.134788 3121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.140764 3121 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:32837
I20260810 21:09:51.140858 3358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:32837 every 8 connection(s)
I20260810 21:09:51.141158 3121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:09:51 dist-test-slave-2kcd krb5kdc[2933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:51.148324 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3121
I20260810 21:09:51.148480 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:51.148706 28117 external_mini_cluster.cc:1428] Setting key 1cfdcc0ce3dbb9ba4671d0e6d44f15e3
I20260810 21:09:51.151074 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.141983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:59261 (local address 127.27.117.126:40085)
0810 21:09:51.142047 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.142050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.142231 (+ 181us) server_negotiation.cc:407] Connection header received
0810 21:09:51.142877 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.142878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.142891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.142944 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:51.143959 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.144471 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.145499 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.145707 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.145779 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.147852 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:51.147853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:51.147853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:51.147861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:51.149637 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.150052 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.150053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.150054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.150114 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.150383 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.150384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.150384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.150512 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:51.150589 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.150872 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.150984 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 21:09:51.151466 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.141938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40085 (local address 127.27.117.65:59261)
0810 21:09:51.142097 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.142146 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.142763 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.143032 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.143036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.143309 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:51.143849 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.143854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.144574 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.144575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.145389 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.145391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.145788 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.145789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.145940 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.146407 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:51.146415 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:51.147747 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:51.149731 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:51.149738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:51.149740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:51.149971 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:51.150191 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:51.150192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:51.150192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:51.150306 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:51.150580 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:51.150588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:51.150733 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.150839 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.151012 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260810 21:09:51.151964 3360 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40085
I20260810 21:09:51.152061 3360 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:51.152242 3360 heartbeater.cc:511] Master 127.27.117.126:40085 requested a full tablet report, sending...
I20260810 21:09:51.152891 3008 ts_manager.cc:194] Registered new tserver with Master: 31565bd9c3b14a9cafa27c80952efbd7 (127.27.117.65:32837)
I20260810 21:09:51.153594 3008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:59261
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:51.173403 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40085
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:51.255152 3366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:51.255374 3366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:51.255415 3366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:51.256544 3366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:51.256609 3366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:51.256654 3366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:51.258229 3366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:51.258502 3366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:51.258730 3366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:51.263564 3376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.263571 3378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.263571 3381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:51.263643 3366 server_base.cc:1034] running on GCE node
I20260810 21:09:51.264004 3366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:51.264246 3366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:51.265405 3366 hybrid_clock.cc:648] HybridClock initialized: now 1786396191265383 us; error 35 us; skew 500 ppm
Aug 10 21:09:51 dist-test-slave-2kcd krb5kdc[2933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:51.267844 3366 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:51.268178 3366 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:51.268251 3366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:51.268400 3366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:51.269441 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "9a9e20132e314eaca8590e154887ccfc"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "a524724e261188972929c39d27c04852"
server_key_iv: "99cfec9d571d77e03c682e01aee42666"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.269805 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "9a9e20132e314eaca8590e154887ccfc"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "a524724e261188972929c39d27c04852"
server_key_iv: "99cfec9d571d77e03c682e01aee42666"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.271062 3366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:51.271762 3396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.271934 3366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:51.272053 3366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "9a9e20132e314eaca8590e154887ccfc"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "a524724e261188972929c39d27c04852"
server_key_iv: "99cfec9d571d77e03c682e01aee42666"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.272128 3366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:51.293244 3366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:51.295111 3366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:51.295279 3366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:51.295523 3366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:51.295856 3366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:51.295913 3366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.295949 3366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:51.296002 3366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.302021 3366 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:37899
I20260810 21:09:51.302062 3582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:37899 every 8 connection(s)
I20260810 21:09:51.302343 3366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:09:51 dist-test-slave-2kcd krb5kdc[2933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:51.308334 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3366
I20260810 21:09:51.308470 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:51.308665 28117 external_mini_cluster.cc:1428] Setting key 8f0e58640c3ba2bd0303e9b70dea6278
I20260810 21:09:51.311676 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.303085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:55291 (local address 127.27.117.126:40085)
0810 21:09:51.303159 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.303162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.303378 (+ 216us) server_negotiation.cc:407] Connection header received
0810 21:09:51.303839 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.303840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.303850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.303899 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:51.304751 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.305311 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.306368 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.306570 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.306624 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.308735 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:51.308736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:51.308737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:51.308746 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:51.310283 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.310693 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.310694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.310694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.310757 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.311028 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.311028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.311029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.311147 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:51.311224 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.311494 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.311582 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:09:51.311872 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.303067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40085 (local address 127.27.117.66:55291)
0810 21:09:51.303254 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.303291 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.303757 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.303956 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.303960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.304205 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:51.304675 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.304680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.305395 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.305396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.306298 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.306301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.306643 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.306643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.306737 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.307162 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:51.307170 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:51.308637 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:51.310369 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:51.310377 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:51.310380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:51.310610 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:51.310828 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:51.310829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:51.310829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:51.310950 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:51.311214 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:51.311221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:51.311361 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.311472 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.311637 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260810 21:09:51.312345 3583 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40085
I20260810 21:09:51.312438 3583 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:51.312601 3583 heartbeater.cc:511] Master 127.27.117.126:40085 requested a full tablet report, sending...
I20260810 21:09:51.312994 3008 ts_manager.cc:194] Registered new tserver with Master: 9a9e20132e314eaca8590e154887ccfc (127.27.117.66:37899)
I20260810 21:09:51.313519 3008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:55291
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:51.333588 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40085
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:51.414228 3592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:51.414417 3592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:51.414438 3592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:51.415493 3592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:51.415524 3592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:51.415549 3592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:51.416951 3592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:51.417119 3592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:51.417309 3592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:09:51.422363 3592 server_base.cc:1034] running on GCE node
W20260810 21:09:51.422329 3607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.422310 3603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.422338 3611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:51.422770 3592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:51.423022 3592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:51.424182 3592 hybrid_clock.cc:648] HybridClock initialized: now 1786396191424174 us; error 38 us; skew 500 ppm
Aug 10 21:09:51 dist-test-slave-2kcd krb5kdc[2933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:51.426771 3592 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:51.427186 3592 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:51.427261 3592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:51.427446 3592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:51.428501 3592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "17862466aeaa4de4ab5f5f7de05aacf6"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "8c300be506f0e73cca030ae3ab18549b"
server_key_iv: "6e5fdd5a97462d9dd7fa1ad00559f573"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.428905 3592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "17862466aeaa4de4ab5f5f7de05aacf6"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "8c300be506f0e73cca030ae3ab18549b"
server_key_iv: "6e5fdd5a97462d9dd7fa1ad00559f573"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.430168 3592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:51.430863 3618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.431044 3592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:51.431130 3592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "17862466aeaa4de4ab5f5f7de05aacf6"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "8c300be506f0e73cca030ae3ab18549b"
server_key_iv: "6e5fdd5a97462d9dd7fa1ad00559f573"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.431182 3592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:51.439553 3592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:51.441483 3592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:51.441656 3592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:51.441872 3592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:51.442251 3592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:51.442315 3592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.442345 3592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:51.442368 3592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.448228 3592 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:45365
I20260810 21:09:51.448302 3815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:45365 every 8 connection(s)
I20260810 21:09:51.448572 3592 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:09:51 dist-test-slave-2kcd krb5kdc[2933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396191, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:51.457180 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.449290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:36141 (local address 127.27.117.126:40085)
0810 21:09:51.449352 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.449355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.449571 (+ 216us) server_negotiation.cc:407] Connection header received
0810 21:09:51.450017 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.450018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.450028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.450070 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:51.450899 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.451389 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.452310 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.452491 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.452546 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.454621 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:51.454622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:51.454623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:51.454629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:51.456010 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.456340 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.456341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.456341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.456393 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.456596 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.456596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.456596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.456695 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:51.456745 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.457051 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.457116 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 21:09:51.457352 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.449270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40085 (local address 127.27.117.67:36141)
0810 21:09:51.449446 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.449487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.449924 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.450118 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.450121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.450367 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:51.450828 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.450833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.451475 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.451476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.452240 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.452242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.452561 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.452562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.452655 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.453126 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:51.453134 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:51.454536 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:51.456082 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:51.456087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:51.456089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:51.456279 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:51.456450 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:51.456451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:51.456451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:51.456544 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:51.456749 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:51.456753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:51.456911 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.457021 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.457136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260810 21:09:51.457944 3816 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40085
I20260810 21:09:51.457967 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3592
I20260810 21:09:51.458065 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:51.458065 3816 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:51.458220 3816 heartbeater.cc:511] Master 127.27.117.126:40085 requested a full tablet report, sending...
I20260810 21:09:51.458272 28117 external_mini_cluster.cc:1428] Setting key a61a21cf2cdacd16e02920c981327eb1
I20260810 21:09:51.458664 3007 ts_manager.cc:194] Registered new tserver with Master: 17862466aeaa4de4ab5f5f7de05aacf6 (127.27.117.67:45365)
I20260810 21:09:51.459136 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:09:51.459167 3007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:36141
I20260810 21:09:51.465809 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.460552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.27.117.126:40085)
0810 21:09:51.460588 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.460590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.460710 (+ 120us) server_negotiation.cc:407] Connection header received
0810 21:09:51.460763 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.460764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.460771 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.460868 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:51.461363 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.461809 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.462476 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.462642 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.462690 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.463821 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:51.463822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:51.463822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:51.463827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:51.464950 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.465237 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.465238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.465238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.465283 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:51.465434 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:51.465434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:51.465435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:51.465557 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:51.465630 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.465706 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.465758 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:09:51.467248 3007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:51.468721 3007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:51.475142 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.471243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.27.117.65:32837)
0810 21:09:51.471427 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.471431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.471443 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:09:51.471484 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.471486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.471500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.471585 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:51.471714 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.472451 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.473987 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.474686 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474778 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.474835 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.475008 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.475059 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260810 21:09:51.475142 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.471101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32837 (local address 127.0.0.1:57168)
0810 21:09:51.471283 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.471294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.471345 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.471583 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.471584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.471586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:51.471649 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.471651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.472558 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.472559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.473888 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.473890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.474780 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.474877 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.474898 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.474988 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.475042 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":42,"threads_started":1}
I20260810 21:09:51.475157 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.470735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45365 (local address 127.0.0.1:47728)
0810 21:09:51.470840 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.470876 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.470933 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.471253 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.471257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.471259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:51.471320 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.471321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.472854 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.472855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.473939 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.473941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.474881 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.474944 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.474949 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.475010 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.475053 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
I20260810 21:09:51.475418 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.471321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.27.117.66:37899)
0810 21:09:51.471505 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.471509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.471518 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:51.471557 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.471559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.471575 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.471645 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:51.471869 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.472958 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474219 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.474920 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474985 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.475037 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.475283 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.475334 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260810 21:09:51.475590 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.470943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:37899 (local address 127.0.0.1:60970)
0810 21:09:51.471059 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.471070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.471119 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.471734 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.471735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.471737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:51.471803 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.471804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.473057 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.473058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.474149 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.474152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.475069 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.475070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.475137 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.475140 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.475188 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.475525 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260810 21:09:51.476081 3243 tablet_service.cc:1467] Processing CreateTablet for tablet 932b8ed425ec484ebe2fd22a96040d04 (DEFAULT_TABLE table=test-table [id=a7342464ab6548c29cdadb600daaa395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:51.476269 3478 tablet_service.cc:1467] Processing CreateTablet for tablet 932b8ed425ec484ebe2fd22a96040d04 (DEFAULT_TABLE table=test-table [id=a7342464ab6548c29cdadb600daaa395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:51.476358 3243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932b8ed425ec484ebe2fd22a96040d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:51.476506 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932b8ed425ec484ebe2fd22a96040d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:51.477198 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.470799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.27.117.67:45365)
0810 21:09:51.470954 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.470959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.471013 (+ 54us) server_negotiation.cc:407] Connection header received
0810 21:09:51.471052 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.471054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.471070 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.471145 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:51.471410 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.472725 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474034 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.474727 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.474795 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.474845 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.477076 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.477123 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260810 21:09:51.477878 3699 tablet_service.cc:1467] Processing CreateTablet for tablet 932b8ed425ec484ebe2fd22a96040d04 (DEFAULT_TABLE table=test-table [id=a7342464ab6548c29cdadb600daaa395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:51.478204 3699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932b8ed425ec484ebe2fd22a96040d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:51.479379 3838 tablet_bootstrap.cc:492] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Bootstrap starting.
I20260810 21:09:51.479990 3838 tablet_bootstrap.cc:654] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:51.480530 3838 log.cc:824] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:51.481076 3839 tablet_bootstrap.cc:492] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: Bootstrap starting.
I20260810 21:09:51.481353 3838 tablet_bootstrap.cc:492] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: No bootstrap required, opened a new log
I20260810 21:09:51.481402 3842 tablet_bootstrap.cc:492] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Bootstrap starting.
I20260810 21:09:51.481436 3838 ts_tablet_manager.cc:1397] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:51.481685 3839 tablet_bootstrap.cc:654] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:51.482017 3842 tablet_bootstrap.cc:654] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:51.482254 3839 log.cc:824] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:51.482625 3842 log.cc:824] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:51.483079 3839 tablet_bootstrap.cc:492] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: No bootstrap required, opened a new log
I20260810 21:09:51.483146 3839 ts_tablet_manager.cc:1397] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:51.483390 3842 tablet_bootstrap.cc:492] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: No bootstrap required, opened a new log
I20260810 21:09:51.483474 3842 ts_tablet_manager.cc:1397] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:51.483608 3838 raft_consensus.cc:348] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.483773 3838 raft_consensus.cc:374] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:51.483819 3838 raft_consensus.cc:729] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9e20132e314eaca8590e154887ccfc, State: Initialized, Role: FOLLOWER
I20260810 21:09:51.483932 3838 consensus_queue.cc:260] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.484215 3838 ts_tablet_manager.cc:1428] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:09:51.484340 3583 heartbeater.cc:503] Master 127.27.117.126:40085 was elected leader, sending a full tablet report...
I20260810 21:09:51.485148 3839 raft_consensus.cc:348] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.485319 3839 raft_consensus.cc:374] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:51.485373 3839 raft_consensus.cc:729] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31565bd9c3b14a9cafa27c80952efbd7, State: Initialized, Role: FOLLOWER
I20260810 21:09:51.485492 3839 consensus_queue.cc:260] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.485530 3842 raft_consensus.cc:348] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.485702 3842 raft_consensus.cc:374] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:51.485756 3842 raft_consensus.cc:729] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17862466aeaa4de4ab5f5f7de05aacf6, State: Initialized, Role: FOLLOWER
I20260810 21:09:51.485809 3839 ts_tablet_manager.cc:1428] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:09:51.485836 3360 heartbeater.cc:503] Master 127.27.117.126:40085 was elected leader, sending a full tablet report...
I20260810 21:09:51.485890 3842 consensus_queue.cc:260] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.486187 3842 ts_tablet_manager.cc:1428] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:09:51.486199 3816 heartbeater.cc:503] Master 127.27.117.126:40085 was elected leader, sending a full tablet report...
W20260810 21:09:51.553052 3584 tablet.cc:2369] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:09:51.641950 3361 tablet.cc:2369] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:51.667205 3845 raft_consensus.cc:482] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:51.667332 3845 raft_consensus.cc:504] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.667672 3845 leader_election.cc:288] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17862466aeaa4de4ab5f5f7de05aacf6 (127.27.117.67:45365), 31565bd9c3b14a9cafa27c80952efbd7 (127.27.117.65:32837)
I20260810 21:09:51.671604 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.668028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:41253 (local address 127.27.117.67:45365)
0810 21:09:51.668099 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.668102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.668114 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:09:51.668159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.668160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.668170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.668234 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:51.668403 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.669200 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.670651 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.671253 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.671318 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.671325 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.671491 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.671530 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:09:51.671604 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.668129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:50269 (local address 127.27.117.65:32837)
0810 21:09:51.668175 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.668177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.668307 (+ 130us) server_negotiation.cc:407] Connection header received
0810 21:09:51.668338 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.668339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.668346 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.668417 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:51.668586 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.669282 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.670650 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.671233 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.671316 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.671325 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.671491 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.671533 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:09:51.671614 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.667891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45365 (local address 127.27.117.66:41253)
0810 21:09:51.667969 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.667982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.668066 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.668245 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.668246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.668248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:51.668330 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.668332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.669283 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.669284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.670557 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.670560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.671328 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.671329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.671416 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.671419 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.671470 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.671514 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":5}
I20260810 21:09:51.671622 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.668033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32837 (local address 127.27.117.66:50269)
0810 21:09:51.668223 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.668234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:51.668290 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:51.668436 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:51.668437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:51.668438 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:51.668504 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.668505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.669380 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.669381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.670557 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:51.670560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.671328 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.671329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:51.671406 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.671410 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:51.671481 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 21:09:51.671514 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":8,"thread_start_us":123,"threads_started":1}
I20260810 21:09:51.672062 3733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932b8ed425ec484ebe2fd22a96040d04" candidate_uuid: "9a9e20132e314eaca8590e154887ccfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" is_pre_election: true
I20260810 21:09:51.672042 3277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932b8ed425ec484ebe2fd22a96040d04" candidate_uuid: "9a9e20132e314eaca8590e154887ccfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31565bd9c3b14a9cafa27c80952efbd7" is_pre_election: true
I20260810 21:09:51.672215 3277 raft_consensus.cc:2454] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9e20132e314eaca8590e154887ccfc in term 0.
I20260810 21:09:51.672215 3733 raft_consensus.cc:2454] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a9e20132e314eaca8590e154887ccfc in term 0.
I20260810 21:09:51.672456 3399 leader_election.cc:302] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31565bd9c3b14a9cafa27c80952efbd7, 9a9e20132e314eaca8590e154887ccfc; no voters:
I20260810 21:09:51.672593 3845 raft_consensus.cc:2781] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:51.672652 3845 raft_consensus.cc:482] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:51.672680 3845 raft_consensus.cc:3037] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:51.673377 3845 raft_consensus.cc:504] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.673534 3845 leader_election.cc:288] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [CANDIDATE]: Term 1 election: Requested vote from peers 17862466aeaa4de4ab5f5f7de05aacf6 (127.27.117.67:45365), 31565bd9c3b14a9cafa27c80952efbd7 (127.27.117.65:32837)
I20260810 21:09:51.673671 3733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932b8ed425ec484ebe2fd22a96040d04" candidate_uuid: "9a9e20132e314eaca8590e154887ccfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17862466aeaa4de4ab5f5f7de05aacf6"
I20260810 21:09:51.673754 3277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932b8ed425ec484ebe2fd22a96040d04" candidate_uuid: "9a9e20132e314eaca8590e154887ccfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31565bd9c3b14a9cafa27c80952efbd7"
I20260810 21:09:51.673774 3733 raft_consensus.cc:3037] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:51.673851 3277 raft_consensus.cc:3037] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:51.674575 3733 raft_consensus.cc:2454] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9e20132e314eaca8590e154887ccfc in term 1.
I20260810 21:09:51.674635 3277 raft_consensus.cc:2454] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a9e20132e314eaca8590e154887ccfc in term 1.
I20260810 21:09:51.674799 3399 leader_election.cc:302] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17862466aeaa4de4ab5f5f7de05aacf6, 9a9e20132e314eaca8590e154887ccfc; no voters:
I20260810 21:09:51.674911 3845 raft_consensus.cc:2781] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:51.675102 3845 raft_consensus.cc:686] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 1 LEADER]: Becoming Leader. State: Replica: 9a9e20132e314eaca8590e154887ccfc, State: Running, Role: LEADER
I20260810 21:09:51.675222 3845 consensus_queue.cc:237] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } }
I20260810 21:09:51.676071 3007 catalog_manager.cc:5495] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a9e20132e314eaca8590e154887ccfc (127.27.117.66). New cstate: current_term: 1 leader_uuid: "9a9e20132e314eaca8590e154887ccfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9e20132e314eaca8590e154887ccfc" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 37899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 } health_report { overall_health: UNKNOWN } } }
W20260810 21:09:51.699149 3817 tablet.cc:2369] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:51.703194 3837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.700835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.27.117.66:37899)
0810 21:09:51.700891 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.700893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.700903 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:51.700947 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.700948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.700981 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.701061 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:51.701212 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.701662 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.702402 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:51.702589 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:51.702666 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:51.702845 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:51.702948 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:51.703050 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.703070 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.703089 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 21:09:51.705003 3277 raft_consensus.cc:1261] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 1 FOLLOWER]: Refusing update from remote peer 9a9e20132e314eaca8590e154887ccfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:51.705003 3733 raft_consensus.cc:1261] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 1 FOLLOWER]: Refusing update from remote peer 9a9e20132e314eaca8590e154887ccfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:51.705344 3845 consensus_queue.cc:1035] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "31565bd9c3b14a9cafa27c80952efbd7" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:51.705518 3850 consensus_queue.cc:1035] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "17862466aeaa4de4ab5f5f7de05aacf6" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:51.707744 3857 mvcc.cc:205] Tried to move back new op lower bound from 7317078801220739072 to 7317078801102233600. Current Snapshot: MvccSnapshot[applied={T|T < 7317078801220739072}]
I20260810 21:09:51.708143 3856 mvcc.cc:205] Tried to move back new op lower bound from 7317078801220739072 to 7317078801102233600. Current Snapshot: MvccSnapshot[applied={T|T < 7317078801220739072}]
I20260810 21:09:51.707602 3855 mvcc.cc:205] Tried to move back new op lower bound from 7317078801220739072 to 7317078801102233600. Current Snapshot: MvccSnapshot[applied={T|T < 7317078801220739072}]
I20260810 21:09:51.710278 3007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:51.710382 3007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:51.711099 3007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f433a469ad4404d8919763d058aaa14: Sending DeleteTablet for 3 replicas of tablet 932b8ed425ec484ebe2fd22a96040d04
I20260810 21:09:51.711400 3699 tablet_service.cc:1514] Processing DeleteTablet for tablet 932b8ed425ec484ebe2fd22a96040d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:51 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47728
I20260810 21:09:51.711493 3478 tablet_service.cc:1514] Processing DeleteTablet for tablet 932b8ed425ec484ebe2fd22a96040d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:51 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60970
I20260810 21:09:51.711632 3243 tablet_service.cc:1514] Processing DeleteTablet for tablet 932b8ed425ec484ebe2fd22a96040d04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:51 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:57168
I20260810 21:09:51.711730 3880 tablet_replica.cc:330] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: stopping tablet replica
I20260810 21:09:51.711824 3880 raft_consensus.cc:2229] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:51.711855 3881 tablet_replica.cc:330] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7: stopping tablet replica
I20260810 21:09:51.711927 3880 raft_consensus.cc:2258] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:51.711948 3881 raft_consensus.cc:2229] T 932b8ed425ec484ebe2fd22a96040d04 P 31565bd9c3b14a9cafa27c80952efbd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:51.711974 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3121
I20260810 21:09:51.715924 3880 ts_tablet_manager.cc:1905] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:51.718189 3880 ts_tablet_manager.cc:1918] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:51.718269 3880 log.cc:1196] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/932b8ed425ec484ebe2fd22a96040d04
I20260810 21:09:51.718619 3880 ts_tablet_manager.cc:1939] T 932b8ed425ec484ebe2fd22a96040d04 P 9a9e20132e314eaca8590e154887ccfc: Deleting consensus metadata
I20260810 21:09:51.719077 2985 catalog_manager.cc:4841] TS 9a9e20132e314eaca8590e154887ccfc (127.27.117.66:37899): tablet 932b8ed425ec484ebe2fd22a96040d04 (table test-table [id=a7342464ab6548c29cdadb600daaa395]) successfully deleted
I20260810 21:09:51.723400 3876 tablet_replica.cc:330] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: stopping tablet replica
I20260810 21:09:51.723520 3876 raft_consensus.cc:2229] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:51.723635 3876 raft_consensus.cc:2258] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:51.724169 3876 ts_tablet_manager.cc:1905] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:51.725831 3876 ts_tablet_manager.cc:1918] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:51.725910 3876 log.cc:1196] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/932b8ed425ec484ebe2fd22a96040d04
I20260810 21:09:51.726177 3876 ts_tablet_manager.cc:1939] T 932b8ed425ec484ebe2fd22a96040d04 P 17862466aeaa4de4ab5f5f7de05aacf6: Deleting consensus metadata
I20260810 21:09:51.726584 2985 catalog_manager.cc:4841] TS 17862466aeaa4de4ab5f5f7de05aacf6 (127.27.117.67:45365): tablet 932b8ed425ec484ebe2fd22a96040d04 (table test-table [id=a7342464ab6548c29cdadb600daaa395]) successfully deleted
W20260810 21:09:51.729408 2985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:32837 (error 108)
I20260810 21:09:51.729936 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.729794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:32837 (local address 127.0.0.1:57174)
0810 21:09:51.729856 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:51.729897 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:32837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 21:09:51.730013 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3366
W20260810 21:09:51.732640 2985 catalog_manager.cc:4584] TS 31565bd9c3b14a9cafa27c80952efbd7 (127.27.117.65:32837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 932b8ed425ec484ebe2fd22a96040d04: Network error: Client connection negotiation failed: client connection to 127.27.117.65:32837: connect: Connection refused (error 111)
I20260810 21:09:51.737319 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3592
I20260810 21:09:51.758579 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2950
2026-08-10T21:09:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2341 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T21:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:51Z Disabled control of system clock
I20260810 21:09:51.791036 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45605
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 21:09:51.871718 3888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:51.871905 3888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:51.871927 3888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:51.873042 3888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:51.873080 3888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:51.873092 3888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:51.873104 3888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:51.873155 3888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:51.874434 3888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:45605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:51.874679 3888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:51.874976 3888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:51.880028 3900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:51.880020 3899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:51.880048 3888 server_base.cc:1034] running on GCE node
W20260810 21:09:51.880019 3903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:51.880457 3888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:51.880847 3888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:51.882006 3888 hybrid_clock.cc:648] HybridClock initialized: now 1786396191881983 us; error 38 us; skew 500 ppm
I20260810 21:09:51.886837 3888 webserver.cc:460] Webserver started at http://127.27.117.126:45107/ using document root <none> and password file <none>
I20260810 21:09:51.887192 3888 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:51.887261 3888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:51.887427 3888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:51.888474 3888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "fc5486a4023e4024914dae29d13d294b"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "9384ccd5189453488bd3cc3d5b336178"
server_key_iv: "373498a3704117794ecc8995b389e984"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.888868 3888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "fc5486a4023e4024914dae29d13d294b"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "9384ccd5189453488bd3cc3d5b336178"
server_key_iv: "373498a3704117794ecc8995b389e984"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.890136 3888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:51.890844 3918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:51.891055 3888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:51.891150 3888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "fc5486a4023e4024914dae29d13d294b"
format_stamp: "Formatted at 2026-08-10 21:09:51 on dist-test-slave-2kcd"
server_key: "9384ccd5189453488bd3cc3d5b336178"
server_key_iv: "373498a3704117794ecc8995b389e984"
server_key_version: "encryptionkey@0"
I20260810 21:09:51.891203 3888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:51.894438 3888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:51.894685 3888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:51.894776 3888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:51.898828 3888 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:45605
I20260810 21:09:51.898891 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:45605 every 8 connection(s)
I20260810 21:09:51.899199 3888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:51.899863 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:51.902673 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b: Bootstrap starting.
I20260810 21:09:51.903239 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:51.903540 4008 log.cc:824] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:51.904177 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b: No bootstrap required, opened a new log
I20260810 21:09:51.905216 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3888
I20260810 21:09:51.905347 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:51.905515 28117 external_mini_cluster.cc:1428] Setting key b9aee6ff32be7962a1f9e61771194b52
I20260810 21:09:51.906309 4008 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5486a4023e4024914dae29d13d294b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45605 } }
I20260810 21:09:51.906476 4008 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:51.906512 4008 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5486a4023e4024914dae29d13d294b, State: Initialized, Role: FOLLOWER
I20260810 21:09:51.906646 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5486a4023e4024914dae29d13d294b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45605 } }
I20260810 21:09:51.906728 4008 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:51.906765 4008 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:51.906816 4008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:51.907442 4008 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5486a4023e4024914dae29d13d294b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45605 } }
I20260810 21:09:51.907557 4008 leader_election.cc:302] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5486a4023e4024914dae29d13d294b; no voters:
I20260810 21:09:51.907723 4008 leader_election.cc:288] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:51.907788 4015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:51.907946 4015 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [term 1 LEADER]: Becoming Leader. State: Replica: fc5486a4023e4024914dae29d13d294b, State: Running, Role: LEADER
I20260810 21:09:51.908000 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:51.908039 4015 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5486a4023e4024914dae29d13d294b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45605 } }
I20260810 21:09:51.908246 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:51.906062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.27.117.126:45605)
0810 21:09:51.906304 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:51.906312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:51.906358 (+ 46us) server_negotiation.cc:407] Connection header received
0810 21:09:51.906462 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:51.906468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:51.906521 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:51.906664 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:51.907362 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:51.907365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:51.907365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:51.907388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:51.907434 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:51.907449 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:51.907531 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:51.907621 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 21:09:51.907788 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260810 21:09:51.908355 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc5486a4023e4024914dae29d13d294b. Latest consensus state: current_term: 1 leader_uuid: "fc5486a4023e4024914dae29d13d294b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5486a4023e4024914dae29d13d294b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45605 } } }
I20260810 21:09:51.908418 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:51.908517 4021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc5486a4023e4024914dae29d13d294b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5486a4023e4024914dae29d13d294b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45605 } } }
I20260810 21:09:51.908648 4021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:51.908883 4028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:51.909824 4028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:51.911167 4028 catalog_manager.cc:1347] Generated new cluster ID: 5926955858824da8ab9fa20cfd8fef8c
I20260810 21:09:51.911219 4028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:51.931593 4028 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:51.931771 4028 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:51.943233 4028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b: Generated new TSK 0
I20260810 21:09:51.943432 4028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:09:51.945423 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45605
--builtin_ntp_servers=127.27.117.84:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 21:09:52.032076 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:52.032274 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:52.032296 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:52.033504 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:52.033546 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:52.033573 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:52.034924 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:52.035172 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:52.035481 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:52.040367 4060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.040501 4050 server_base.cc:1034] running on GCE node
W20260810 21:09:52.040393 4058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:52.040606 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.040892 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:52.041157 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:52.042323 4050 hybrid_clock.cc:648] HybridClock initialized: now 1786396192042317 us; error 40 us; skew 500 ppm
I20260810 21:09:52.043505 4050 webserver.cc:460] Webserver started at http://127.27.117.65:39715/ using document root <none> and password file <none>
I20260810 21:09:52.043757 4050 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:52.043824 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:52.043980 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:52.045094 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "528176845bdc424ea737601061819943"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "fbac7b61ed5c66021192a484094fe9d1"
server_key_iv: "72d7c24ddf8930582de7a4412ddcda19"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.045451 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "528176845bdc424ea737601061819943"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "fbac7b61ed5c66021192a484094fe9d1"
server_key_iv: "72d7c24ddf8930582de7a4412ddcda19"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.046742 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:52.047427 4074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.047602 4050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:52.047703 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "528176845bdc424ea737601061819943"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "fbac7b61ed5c66021192a484094fe9d1"
server_key_iv: "72d7c24ddf8930582de7a4412ddcda19"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.047780 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:52.051043 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:52.051287 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:52.051424 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:52.051623 4050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:52.051923 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:52.051990 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.052030 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:52.052057 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.057704 4050 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:45597
I20260810 21:09:52.057786 4267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:45597 every 8 connection(s)
I20260810 21:09:52.058090 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:52.061002 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.058783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:54335 (local address 127.27.117.126:45605)
0810 21:09:52.058830 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.058832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.059088 (+ 256us) server_negotiation.cc:407] Connection header received
0810 21:09:52.059508 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.059510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.059516 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.059569 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.060734 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.060735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.060735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.060738 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.060749 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.060753 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.060833 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.060934 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.060948 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:09:52.061170 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.058750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45605 (local address 127.27.117.65:54335)
0810 21:09:52.058926 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.058985 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.059426 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.059618 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.059621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.059629 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.060605 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.060606 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.060632 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.060659 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.060863 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.060866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.060869 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.060924 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.060971 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260810 21:09:52.061383 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 4050
I20260810 21:09:52.061503 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:52.061700 28117 external_mini_cluster.cc:1428] Setting key d186514bc7764c283bb88eae2365c3fb
I20260810 21:09:52.061762 4270 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45605
I20260810 21:09:52.061859 4270 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:52.062014 4270 heartbeater.cc:511] Master 127.27.117.126:45605 requested a full tablet report, sending...
I20260810 21:09:52.062507 3947 ts_manager.cc:194] Registered new tserver with Master: 528176845bdc424ea737601061819943 (127.27.117.65:45597)
I20260810 21:09:52.062701 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45605
--builtin_ntp_servers=127.27.117.84:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 21:09:52.144737 4277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:52.145037 4277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:52.145061 4277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:52.146309 4277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:52.146351 4277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:52.146376 4277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:52.147653 4277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:52.147888 4277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:52.148146 4277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:52.153168 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:52.153208 4288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.153395 4277 server_base.cc:1034] running on GCE node
W20260810 21:09:52.153429 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.153661 4277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:52.153895 4277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:52.155047 4277 hybrid_clock.cc:648] HybridClock initialized: now 1786396192155029 us; error 40 us; skew 500 ppm
I20260810 21:09:52.156291 4277 webserver.cc:460] Webserver started at http://127.27.117.66:44173/ using document root <none> and password file <none>
I20260810 21:09:52.156533 4277 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:52.156579 4277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:52.156744 4277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:52.157835 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "c1a7d52281c243aab07fe511e3faf5d1"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "4598f27815907391d61ba4a42701f296"
server_key_iv: "cd1c4438899f6365d7894cc19672de15"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.158252 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "c1a7d52281c243aab07fe511e3faf5d1"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "4598f27815907391d61ba4a42701f296"
server_key_iv: "cd1c4438899f6365d7894cc19672de15"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.159552 4277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:52.160264 4300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.160437 4277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:52.160537 4277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "c1a7d52281c243aab07fe511e3faf5d1"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "4598f27815907391d61ba4a42701f296"
server_key_iv: "cd1c4438899f6365d7894cc19672de15"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.160610 4277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:52.163926 4277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:52.164181 4277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:52.164318 4277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:52.164543 4277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:52.164899 4277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:52.164934 4277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.164986 4277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:52.165014 4277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.170755 4277 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36931
I20260810 21:09:52.170799 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36931 every 8 connection(s)
I20260810 21:09:52.171137 4277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:52.173941 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.171868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:53835 (local address 127.27.117.126:45605)
0810 21:09:52.171951 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.171953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.172138 (+ 185us) server_negotiation.cc:407] Connection header received
0810 21:09:52.172558 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.172559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.172564 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.172616 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.173769 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.173770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.173770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.173774 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.173786 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.173789 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.173833 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.173879 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.173890 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 21:09:52.174129 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.171831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45605 (local address 127.27.117.66:53835)
0810 21:09:52.172021 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.172059 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.172479 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.172665 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.172668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.172676 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.173646 (+ 970us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.173647 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.173672 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.173697 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.173832 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.173834 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.173837 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.173886 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.173930 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260810 21:09:52.174597 4486 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45605
I20260810 21:09:52.174675 4486 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:52.174839 4486 heartbeater.cc:511] Master 127.27.117.126:45605 requested a full tablet report, sending...
I20260810 21:09:52.175209 3947 ts_manager.cc:194] Registered new tserver with Master: c1a7d52281c243aab07fe511e3faf5d1 (127.27.117.66:36931)
I20260810 21:09:52.178360 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 4277
I20260810 21:09:52.178445 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:52.178582 28117 external_mini_cluster.cc:1428] Setting key 6fb2d8523fba59bbfc318e8e0d2bd8bc
I20260810 21:09:52.179433 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45605
--builtin_ntp_servers=127.27.117.84:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 21:09:52.264058 4490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:52.264297 4490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:52.264317 4490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:52.265493 4490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:52.265533 4490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:52.265558 4490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:52.266824 4490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:52.267006 4490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:52.267251 4490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:52.272101 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.272195 4490 server_base.cc:1034] running on GCE node
W20260810 21:09:52.272073 4498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:52.272082 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.272588 4490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:52.272831 4490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:52.273981 4490 hybrid_clock.cc:648] HybridClock initialized: now 1786396192273973 us; error 27 us; skew 500 ppm
I20260810 21:09:52.275305 4490 webserver.cc:460] Webserver started at http://127.27.117.67:41671/ using document root <none> and password file <none>
I20260810 21:09:52.275566 4490 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:52.275614 4490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:52.275785 4490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:52.276887 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "0301d0d26d5c42a4b30fc918adb09490"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "6a5a9570ee88bb8191499afadfe20dbc"
server_key_iv: "21d961ab2b775050da2de9fde2e9b3c2"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.277241 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "0301d0d26d5c42a4b30fc918adb09490"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "6a5a9570ee88bb8191499afadfe20dbc"
server_key_iv: "21d961ab2b775050da2de9fde2e9b3c2"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.278522 4490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:52.279217 4515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.279387 4490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:52.279495 4490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "0301d0d26d5c42a4b30fc918adb09490"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "6a5a9570ee88bb8191499afadfe20dbc"
server_key_iv: "21d961ab2b775050da2de9fde2e9b3c2"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.279575 4490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:52.282899 4490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:52.283151 4490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:52.283294 4490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:52.283509 4490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:52.283824 4490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:52.283900 4490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.283939 4490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:52.283965 4490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.289825 4490 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:39011
I20260810 21:09:52.289875 4703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:39011 every 8 connection(s)
I20260810 21:09:52.290171 4490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:52.292939 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.290854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:42841 (local address 127.27.117.126:45605)
0810 21:09:52.290937 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.290940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.291117 (+ 177us) server_negotiation.cc:407] Connection header received
0810 21:09:52.291514 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.291515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.291519 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.291571 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.292696 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.292697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.292697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.292700 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.292710 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.292713 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.292752 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.292862 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.292877 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 21:09:52.293180 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.290802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45605 (local address 127.27.117.67:42841)
0810 21:09:52.290998 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.291030 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.291434 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.291617 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.291620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.291627 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.292571 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.292572 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.292598 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.292624 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.292751 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.292753 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.292756 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.292872 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.292924 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260810 21:09:52.293627 4704 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45605
I20260810 21:09:52.293730 4704 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:52.293877 4704 heartbeater.cc:511] Master 127.27.117.126:45605 requested a full tablet report, sending...
I20260810 21:09:52.294266 3947 ts_manager.cc:194] Registered new tserver with Master: 0301d0d26d5c42a4b30fc918adb09490 (127.27.117.67:39011)
I20260810 21:09:52.294639 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 4490
I20260810 21:09:52.294777 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:52.294945 28117 external_mini_cluster.cc:1428] Setting key 4070bf5ac4a291abbb63b0d0f5c82796
I20260810 21:09:52.295814 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:09:52.298174 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.297189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.27.117.126:45605)
0810 21:09:52.297240 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.297241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.297379 (+ 138us) server_negotiation.cc:407] Connection header received
0810 21:09:52.297428 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.297428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.297432 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.297480 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.298036 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.298037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.298037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.298039 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.298049 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.298052 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.298086 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.298125 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.298136 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 21:09:52.299752 3947 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:52.301292 3947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:52.305294 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.303343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:39011 (local address 127.0.0.1:42556)
0810 21:09:52.303812 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.303822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.303866 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.304332 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.304332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.304335 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.304809 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.304809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.304825 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.304847 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.305088 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.305096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.305099 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.305185 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.305198 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260810 21:09:52.305400 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.303343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36931 (local address 127.0.0.1:59256)
0810 21:09:52.303631 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.303670 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.303726 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.304038 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.304044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.304050 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.305001 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.305002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.305009 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.305023 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.305278 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.305279 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.305280 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.305309 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.305324 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":55,"threads_started":1}
I20260810 21:09:52.305320 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.303498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.27.117.67:39011)
0810 21:09:52.303705 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.303709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.303870 (+ 161us) server_negotiation.cc:407] Connection header received
0810 21:09:52.303918 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.303921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.303930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.304003 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.304928 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.304934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.304934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.304938 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.304983 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.305004 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.305078 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.305204 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.305220 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260810 21:09:52.305708 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.303688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.27.117.66:36931)
0810 21:09:52.303875 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.303879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.303889 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:52.303920 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.303922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.303929 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.304021 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.305109 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.305116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.305116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.305120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.305152 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.305198 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.305588 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.305613 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.305628 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260810 21:09:52.305989 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.303361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:45597 (local address 127.0.0.1:44752)
0810 21:09:52.303953 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.303963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.304001 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.304445 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.304446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.304448 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.305608 (+ 1160us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.305609 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.305617 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.305630 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.305865 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.305865 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.305866 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.305923 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.305935 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":102,"threads_started":1}
I20260810 21:09:52.306030 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.303621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.27.117.65:45597)
0810 21:09:52.303814 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.303817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.304067 (+ 250us) server_negotiation.cc:407] Connection header received
0810 21:09:52.304098 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.304100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.304109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.304172 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.305723 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.305730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.305731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.305736 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.305774 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.305807 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.305870 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.305929 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.305945 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260810 21:09:52.306154 4590 tablet_service.cc:1467] Processing CreateTablet for tablet b451a096ecf246c882b7fdf5e65c6bc4 (DEFAULT_TABLE table=test-table [id=4a223634546e4028ac9cc272fa00eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:52.306514 4590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b451a096ecf246c882b7fdf5e65c6bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:52.306860 4156 tablet_service.cc:1467] Processing CreateTablet for tablet b451a096ecf246c882b7fdf5e65c6bc4 (DEFAULT_TABLE table=test-table [id=4a223634546e4028ac9cc272fa00eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:52.307170 4156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b451a096ecf246c882b7fdf5e65c6bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:52.309580 4729 tablet_bootstrap.cc:492] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Bootstrap starting.
I20260810 21:09:52.310249 4375 tablet_service.cc:1467] Processing CreateTablet for tablet b451a096ecf246c882b7fdf5e65c6bc4 (DEFAULT_TABLE table=test-table [id=4a223634546e4028ac9cc272fa00eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:52.310463 4729 tablet_bootstrap.cc:654] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:52.310544 4375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b451a096ecf246c882b7fdf5e65c6bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:52.310771 4729 log.cc:824] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:52.311601 4729 tablet_bootstrap.cc:492] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: No bootstrap required, opened a new log
I20260810 21:09:52.311673 4729 ts_tablet_manager.cc:1397] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:52.313102 4735 tablet_bootstrap.cc:492] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Bootstrap starting.
I20260810 21:09:52.313169 4730 tablet_bootstrap.cc:492] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: Bootstrap starting.
I20260810 21:09:52.313757 4730 tablet_bootstrap.cc:654] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:52.313932 4735 tablet_bootstrap.cc:654] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:52.313869 4729 raft_consensus.cc:348] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.314049 4729 raft_consensus.cc:374] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:52.314105 4729 raft_consensus.cc:729] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0301d0d26d5c42a4b30fc918adb09490, State: Initialized, Role: FOLLOWER
I20260810 21:09:52.314108 4730 log.cc:824] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:52.314248 4735 log.cc:824] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:52.314251 4729 consensus_queue.cc:260] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.314513 4729 ts_tablet_manager.cc:1428] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:09:52.314569 4704 heartbeater.cc:503] Master 127.27.117.126:45605 was elected leader, sending a full tablet report...
I20260810 21:09:52.314949 4730 tablet_bootstrap.cc:492] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: No bootstrap required, opened a new log
I20260810 21:09:52.315017 4730 ts_tablet_manager.cc:1397] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:09:52.315151 4735 tablet_bootstrap.cc:492] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: No bootstrap required, opened a new log
I20260810 21:09:52.315217 4735 ts_tablet_manager.cc:1397] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:52.317092 4730 raft_consensus.cc:348] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.317225 4730 raft_consensus.cc:374] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:52.317267 4730 raft_consensus.cc:729] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528176845bdc424ea737601061819943, State: Initialized, Role: FOLLOWER
I20260810 21:09:52.317266 4735 raft_consensus.cc:348] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.317431 4735 raft_consensus.cc:374] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:52.317379 4730 consensus_queue.cc:260] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.317462 4735 raft_consensus.cc:729] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a7d52281c243aab07fe511e3faf5d1, State: Initialized, Role: FOLLOWER
I20260810 21:09:52.317590 4735 consensus_queue.cc:260] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.317660 4730 ts_tablet_manager.cc:1428] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:09:52.317744 4270 heartbeater.cc:503] Master 127.27.117.126:45605 was elected leader, sending a full tablet report...
I20260810 21:09:52.317836 4735 ts_tablet_manager.cc:1428] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:09:52.317867 4486 heartbeater.cc:503] Master 127.27.117.126:45605 was elected leader, sending a full tablet report...
W20260810 21:09:52.421693 4487 tablet.cc:2369] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:52.463867 4739 raft_consensus.cc:482] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:52.464002 4739 raft_consensus.cc:504] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.464432 4739 leader_election.cc:288] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1a7d52281c243aab07fe511e3faf5d1 (127.27.117.66:36931), 528176845bdc424ea737601061819943 (127.27.117.65:45597)
I20260810 21:09:52.466006 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.464859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:58731 (local address 127.27.117.65:45597)
0810 21:09:52.464921 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.464923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.464952 (+ 29us) server_negotiation.cc:407] Connection header received
0810 21:09:52.465027 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.465028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.465032 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.465105 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.465765 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.465767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.465767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.465772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.465796 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.465801 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.465866 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.465915 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.465930 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 21:09:52.466022 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.464717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:45597 (local address 127.27.117.67:58731)
0810 21:09:52.464886 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.464899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.464957 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.465112 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.465113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.465117 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.465660 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.465661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.465667 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.465679 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.465863 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.465864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.465865 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.465916 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.465930 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
I20260810 21:09:52.466079 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.464917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45429 (local address 127.27.117.66:36931)
0810 21:09:52.464985 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.464987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.464995 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:52.465018 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.465019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.465023 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.465099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.465843 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.465845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.465845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.465849 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.465865 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.465870 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.465930 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.465978 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.465993 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:09:52.466442 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.464657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36931 (local address 127.27.117.67:45429)
0810 21:09:52.464716 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.464729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.464822 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.465105 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.465106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.465109 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:52.465729 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:09:52.465729 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:52.465737 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:09:52.465750 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:52.465928 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:52.465929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:52.465929 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 21:09:52.465978 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 21:09:52.465993 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 21:09:52.466426 4181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451a096ecf246c882b7fdf5e65c6bc4" candidate_uuid: "0301d0d26d5c42a4b30fc918adb09490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528176845bdc424ea737601061819943" is_pre_election: true
I20260810 21:09:52.466580 4181 raft_consensus.cc:2454] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0301d0d26d5c42a4b30fc918adb09490 in term 0.
I20260810 21:09:52.466828 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451a096ecf246c882b7fdf5e65c6bc4" candidate_uuid: "0301d0d26d5c42a4b30fc918adb09490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a7d52281c243aab07fe511e3faf5d1" is_pre_election: true
I20260810 21:09:52.466965 4403 raft_consensus.cc:2454] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0301d0d26d5c42a4b30fc918adb09490 in term 0.
I20260810 21:09:52.467258 4519 leader_election.cc:302] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0301d0d26d5c42a4b30fc918adb09490, 528176845bdc424ea737601061819943; no voters:
I20260810 21:09:52.467430 4739 raft_consensus.cc:2781] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:52.467480 4739 raft_consensus.cc:482] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:52.467497 4739 raft_consensus.cc:3037] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:52.468191 4739 raft_consensus.cc:504] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.468328 4739 leader_election.cc:288] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [CANDIDATE]: Term 1 election: Requested vote from peers c1a7d52281c243aab07fe511e3faf5d1 (127.27.117.66:36931), 528176845bdc424ea737601061819943 (127.27.117.65:45597)
I20260810 21:09:52.468483 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451a096ecf246c882b7fdf5e65c6bc4" candidate_uuid: "0301d0d26d5c42a4b30fc918adb09490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a7d52281c243aab07fe511e3faf5d1"
I20260810 21:09:52.468492 4181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451a096ecf246c882b7fdf5e65c6bc4" candidate_uuid: "0301d0d26d5c42a4b30fc918adb09490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528176845bdc424ea737601061819943"
I20260810 21:09:52.468575 4403 raft_consensus.cc:3037] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:52.468575 4181 raft_consensus.cc:3037] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:52.469489 4181 raft_consensus.cc:2454] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0301d0d26d5c42a4b30fc918adb09490 in term 1.
I20260810 21:09:52.469501 4403 raft_consensus.cc:2454] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0301d0d26d5c42a4b30fc918adb09490 in term 1.
I20260810 21:09:52.469694 4519 leader_election.cc:302] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0301d0d26d5c42a4b30fc918adb09490, 528176845bdc424ea737601061819943; no voters:
I20260810 21:09:52.469785 4739 raft_consensus.cc:2781] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:52.469980 4739 raft_consensus.cc:686] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 1 LEADER]: Becoming Leader. State: Replica: 0301d0d26d5c42a4b30fc918adb09490, State: Running, Role: LEADER
I20260810 21:09:52.470078 4739 consensus_queue.cc:237] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } }
I20260810 21:09:52.470933 3945 catalog_manager.cc:5495] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0301d0d26d5c42a4b30fc918adb09490 (127.27.117.67). New cstate: current_term: 1 leader_uuid: "0301d0d26d5c42a4b30fc918adb09490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0301d0d26d5c42a4b30fc918adb09490" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 39011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 } health_report { overall_health: UNKNOWN } } }
I20260810 21:09:52.485898 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.485030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.27.117.67:39011)
0810 21:09:52.485071 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.485073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.485090 (+ 17us) server_negotiation.cc:407] Connection header received
0810 21:09:52.485153 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.485154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.485163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.485207 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.485761 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.485762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.485762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.485765 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.485774 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.485777 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.485815 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.485847 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.485857 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:09:52.487531 4403 raft_consensus.cc:1261] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 1 FOLLOWER]: Refusing update from remote peer 0301d0d26d5c42a4b30fc918adb09490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:52.487531 4181 raft_consensus.cc:1261] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 1 FOLLOWER]: Refusing update from remote peer 0301d0d26d5c42a4b30fc918adb09490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:52.487848 4739 consensus_queue.cc:1035] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1a7d52281c243aab07fe511e3faf5d1" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:52.487989 4748 consensus_queue.cc:1035] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528176845bdc424ea737601061819943" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 45597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:52.490092 4752 mvcc.cc:205] Tried to move back new op lower bound from 7317078804426559488 to 7317078804358107136. Current Snapshot: MvccSnapshot[applied={T|T < 7317078804426559488}]
I20260810 21:09:52.491292 4754 mvcc.cc:205] Tried to move back new op lower bound from 7317078804426559488 to 7317078804358107136. Current Snapshot: MvccSnapshot[applied={T|T < 7317078804426559488 or (T in {7317078804426559488})}]
I20260810 21:09:52.492219 3945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:52.492316 3945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:52.493026 3945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc5486a4023e4024914dae29d13d294b: Sending DeleteTablet for 3 replicas of tablet b451a096ecf246c882b7fdf5e65c6bc4
I20260810 21:09:52.493315 4375 tablet_service.cc:1514] Processing DeleteTablet for tablet b451a096ecf246c882b7fdf5e65c6bc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:52 UTC) from {username='slave'} at 127.0.0.1:59256
I20260810 21:09:52.493352 4590 tablet_service.cc:1514] Processing DeleteTablet for tablet b451a096ecf246c882b7fdf5e65c6bc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:52 UTC) from {username='slave'} at 127.0.0.1:42556
I20260810 21:09:52.493494 4156 tablet_service.cc:1514] Processing DeleteTablet for tablet b451a096ecf246c882b7fdf5e65c6bc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:52 UTC) from {username='slave'} at 127.0.0.1:44752
I20260810 21:09:52.493703 4766 tablet_replica.cc:330] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: stopping tablet replica
I20260810 21:09:52.493703 4767 tablet_replica.cc:330] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943: stopping tablet replica
I20260810 21:09:52.493801 4767 raft_consensus.cc:2229] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:52.493830 4766 raft_consensus.cc:2229] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:52.493873 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 4050
I20260810 21:09:52.493906 4767 raft_consensus.cc:2258] T b451a096ecf246c882b7fdf5e65c6bc4 P 528176845bdc424ea737601061819943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:52.497503 4765 tablet_replica.cc:330] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: stopping tablet replica
I20260810 21:09:52.497606 4765 raft_consensus.cc:2229] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:52.497538 4766 raft_consensus.cc:2258] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:52.497699 4765 raft_consensus.cc:2258] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:52.498142 4766 ts_tablet_manager.cc:1905] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:52.498143 4765 ts_tablet_manager.cc:1905] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:52.500805 4766 ts_tablet_manager.cc:1918] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:52.500891 4766 log.cc:1196] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/b451a096ecf246c882b7fdf5e65c6bc4
I20260810 21:09:52.501184 4766 ts_tablet_manager.cc:1939] T b451a096ecf246c882b7fdf5e65c6bc4 P c1a7d52281c243aab07fe511e3faf5d1: Deleting consensus metadata
I20260810 21:09:52.501652 3923 catalog_manager.cc:4841] TS c1a7d52281c243aab07fe511e3faf5d1 (127.27.117.66:36931): tablet b451a096ecf246c882b7fdf5e65c6bc4 (table test-table [id=4a223634546e4028ac9cc272fa00eaf5]) successfully deleted
I20260810 21:09:52.507505 4765 ts_tablet_manager.cc:1918] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:52.507673 4765 log.cc:1196] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/b451a096ecf246c882b7fdf5e65c6bc4
W20260810 21:09:52.507676 3922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:45597 (error 108)
I20260810 21:09:52.508014 4765 ts_tablet_manager.cc:1939] T b451a096ecf246c882b7fdf5e65c6bc4 P 0301d0d26d5c42a4b30fc918adb09490: Deleting consensus metadata
I20260810 21:09:52.508173 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.508027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:45597 (local address 127.0.0.1:44754)
0810 21:09:52.508090 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.508138 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 21:09:52.508283 3922 catalog_manager.cc:4584] TS 528176845bdc424ea737601061819943 (127.27.117.65:45597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b451a096ecf246c882b7fdf5e65c6bc4: Network error: Client connection negotiation failed: client connection to 127.27.117.65:45597: connect: Connection refused (error 111)
I20260810 21:09:52.508481 3925 catalog_manager.cc:4841] TS 0301d0d26d5c42a4b30fc918adb09490 (127.27.117.67:39011): tablet b451a096ecf246c882b7fdf5e65c6bc4 (table test-table [id=4a223634546e4028ac9cc272fa00eaf5]) successfully deleted
I20260810 21:09:52.508615 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 4277
I20260810 21:09:52.518344 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 4490
I20260810 21:09:52.531603 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3888
2026-08-10T21:09:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (772 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 21:09:52.556497 28117 mini_oidc.cc:152] Starting JWKS server
I20260810 21:09:52.557742 28117 webserver.cc:460] Webserver started at https://127.0.0.1:46065/ using document root <none> and password file <none>
I20260810 21:09:52.557849 28117 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 21:09:52.558018 28117 webserver.cc:460] Webserver started at http://127.0.0.1:43713/ using document root <none> and password file <none>
2026-08-10T21:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:52Z Disabled control of system clock
I20260810 21:09:52.566525 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35661
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46065/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/testchainca.cert with env {}
W20260810 21:09:52.648000 4779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:52.648200 4779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:52.648224 4779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:52.649422 4779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:52.649463 4779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:52.649475 4779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:52.649488 4779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:52.649542 4779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 21:09:52.649561 4779 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 21:09:52.649585 4779 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46065/jwks.json
I20260810 21:09:52.651181 4779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46065/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:52.651448 4779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:52.651690 4779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:52.656486 4788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:52.656477 4787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.656643 4779 server_base.cc:1034] running on GCE node
W20260810 21:09:52.656478 4791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.657784 4779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:52.658084 4779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:52.659227 4779 hybrid_clock.cc:648] HybridClock initialized: now 1786396192659200 us; error 41 us; skew 500 ppm
I20260810 21:09:52.660558 4779 webserver.cc:460] Webserver started at http://127.27.117.126:37235/ using document root <none> and password file <none>
I20260810 21:09:52.660868 4779 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:52.660920 4779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:52.661069 4779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:52.662220 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "347a9a7d0b6348c581e6aed6fcc68ff2"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "6fc42a607d4ebe8eac5b03ec54bf5121"
server_key_iv: "b50379b79322daa7ec8e0f734ff7fcdc"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.662612 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "347a9a7d0b6348c581e6aed6fcc68ff2"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "6fc42a607d4ebe8eac5b03ec54bf5121"
server_key_iv: "b50379b79322daa7ec8e0f734ff7fcdc"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.664059 4779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:52.664755 4802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.664944 4779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:52.665051 4779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "347a9a7d0b6348c581e6aed6fcc68ff2"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "6fc42a607d4ebe8eac5b03ec54bf5121"
server_key_iv: "b50379b79322daa7ec8e0f734ff7fcdc"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.665138 4779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:52.673413 4779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:52.673728 4779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:52.673882 4779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:52.678030 4779 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35661
I20260810 21:09:52.678061 4878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35661 every 8 connection(s)
I20260810 21:09:52.678392 4779 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:52.679386 4879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:52.680985 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 4779
I20260810 21:09:52.681097 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:52.681293 28117 external_mini_cluster.cc:1428] Setting key 45ee004a576494a4867129c67e957b0b
I20260810 21:09:52.682124 4879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2: Bootstrap starting.
I20260810 21:09:52.682787 4879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:52.683136 4879 log.cc:824] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:52.683872 4879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2: No bootstrap required, opened a new log
I20260810 21:09:52.686178 4879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35661 } }
I20260810 21:09:52.686311 4879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:52.686334 4879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347a9a7d0b6348c581e6aed6fcc68ff2, State: Initialized, Role: FOLLOWER
I20260810 21:09:52.686482 4879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35661 } }
I20260810 21:09:52.686553 4879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:52.686578 4879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:52.686619 4879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:52.687239 4879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35661 } }
I20260810 21:09:52.687355 4879 leader_election.cc:302] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347a9a7d0b6348c581e6aed6fcc68ff2; no voters:
I20260810 21:09:52.687431 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.681819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.27.117.126:35661)
0810 21:09:52.682023 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.682031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.682056 (+ 25us) server_negotiation.cc:407] Connection header received
0810 21:09:52.682503 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.682510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.682579 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.682693 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.682986 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.684338 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.685189 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.685862 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.686008 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:52.686716 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.686719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.686720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.686880 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.686914 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.686924 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.686994 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.687082 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.687164 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260810 21:09:52.687515 4879 leader_election.cc:288] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:52.687677 4885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:52.687742 4879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:52.687827 4885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [term 1 LEADER]: Becoming Leader. State: Replica: 347a9a7d0b6348c581e6aed6fcc68ff2, State: Running, Role: LEADER
I20260810 21:09:52.687958 4885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35661 } }
I20260810 21:09:52.688349 4886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347a9a7d0b6348c581e6aed6fcc68ff2. Latest consensus state: current_term: 1 leader_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35661 } } }
I20260810 21:09:52.688398 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:09:52.688383 4884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347a9a7d0b6348c581e6aed6fcc68ff2" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35661 } } }
I20260810 21:09:52.688524 4886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:52.688544 4884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:52.689389 4902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:52.690001 4902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:52.691457 4902 catalog_manager.cc:1347] Generated new cluster ID: 19f91135d41f4bba8935115ed96c4f5e
I20260810 21:09:52.691502 4902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:52.700989 4902 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:52.701469 4902 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:52.707295 4902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 347a9a7d0b6348c581e6aed6fcc68ff2: Generated new TSK 0
I20260810 21:09:52.707471 4902 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 21:09:52.736393 4936 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 21:09:52.736528 4936 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 21:09:52.736720 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.720839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.27.117.126:35661)
0810 21:09:52.720894 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.720896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.721008 (+ 112us) server_negotiation.cc:407] Connection header received
0810 21:09:52.721072 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.721073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.721084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.721155 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0810 21:09:52.721266 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.722025 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.722748 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.723073 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.723142 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:52.723312 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 21:09:52.736501 (+ 13189us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 21:09:52.736654 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 21:09:52.736891 4882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 21:09:52.737180 4932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:52.720738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35661 (local address 127.0.0.1:43900)
0810 21:09:52.720947 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.720955 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.721008 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.721165 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.721165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.721167 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 21:09:52.721215 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:52.721217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.722095 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.722095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:52.722673 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:52.722675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.723158 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.723159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:52.723200 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:52.723218 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 21:09:52.736648 (+ 13430us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 21:09:52.736759 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.117.126:35661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.117.126:35661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 21:09:52.737653 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 4779
2026-08-10T21:09:52Z chronyd exiting
I20260810 21:09:52.783860 28117 test_util.cc:175] -----------------------------------------------
I20260810 21:09:52.783931 28117 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786396166878326-28117-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 21:09:52.784696 28117 mini_oidc.cc:152] Starting JWKS server
I20260810 21:09:52.785387 28117 webserver.cc:460] Webserver started at https://127.0.0.1:33059/ using document root <none> and password file <none>
I20260810 21:09:52.785429 28117 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 21:09:52.785538 28117 webserver.cc:460] Webserver started at http://127.0.0.1:33989/ using document root <none> and password file <none>
2026-08-10T21:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:52Z Disabled control of system clock
I20260810 21:09:52.794099 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38703
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38703
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/testchainca.cert with env {}
W20260810 21:09:52.896479 4945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:52.896687 4945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:52.896708 4945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:52.897878 4945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:52.897920 4945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:52.897934 4945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:52.897948 4945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:52.897959 4945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 21:09:52.897971 4945 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 21:09:52.897990 4945 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33059/jwks.json
I20260810 21:09:52.899435 4945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:52.899627 4945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:52.899786 4945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:52.904510 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:52.904619 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:52.904548 4960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:52.904664 4945 server_base.cc:1034] running on GCE node
I20260810 21:09:52.904958 4945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:52.905258 4945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:52.906399 4945 hybrid_clock.cc:648] HybridClock initialized: now 1786396192906388 us; error 33 us; skew 500 ppm
I20260810 21:09:52.907627 4945 webserver.cc:460] Webserver started at http://127.27.117.126:43565/ using document root <none> and password file <none>
I20260810 21:09:52.907886 4945 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:52.907932 4945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:52.908093 4945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:52.909240 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "7f23ca9b138647be9d7f8e8c8f0df703"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "086c10ddda6c44df3516dc98cb7ada44"
server_key_iv: "72fd28f03c1e123d6597715463c121c4"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.909648 4945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "7f23ca9b138647be9d7f8e8c8f0df703"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "086c10ddda6c44df3516dc98cb7ada44"
server_key_iv: "72fd28f03c1e123d6597715463c121c4"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.911037 4945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:52.911759 4967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:52.911944 4945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:52.912014 4945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "7f23ca9b138647be9d7f8e8c8f0df703"
format_stamp: "Formatted at 2026-08-10 21:09:52 on dist-test-slave-2kcd"
server_key: "086c10ddda6c44df3516dc98cb7ada44"
server_key_iv: "72fd28f03c1e123d6597715463c121c4"
server_key_version: "encryptionkey@0"
I20260810 21:09:52.912101 4945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:52.924116 4945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:52.924379 4945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:52.924525 4945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:52.928383 4945 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:38703
I20260810 21:09:52.928419 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:38703 every 8 connection(s)
I20260810 21:09:52.928845 4945 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:52.929525 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:52.932147 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703: Bootstrap starting.
I20260810 21:09:52.932729 5059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:52.933056 5059 log.cc:824] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:52.933732 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703: No bootstrap required, opened a new log
I20260810 21:09:52.935326 5059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38703 } }
I20260810 21:09:52.935461 5059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:52.935518 5059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f23ca9b138647be9d7f8e8c8f0df703, State: Initialized, Role: FOLLOWER
I20260810 21:09:52.935707 5059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38703 } }
I20260810 21:09:52.935791 5059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:52.935840 5059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:52.935894 5059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:52.936419 5059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38703 } }
I20260810 21:09:52.936530 5059 leader_election.cc:302] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f23ca9b138647be9d7f8e8c8f0df703; no voters:
I20260810 21:09:52.936715 5059 leader_election.cc:288] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:52.936749 5063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:52.936918 5063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [term 1 LEADER]: Becoming Leader. State: Replica: 7f23ca9b138647be9d7f8e8c8f0df703, State: Running, Role: LEADER
I20260810 21:09:52.937000 5059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:52.937071 5063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38703 } }
I20260810 21:09:52.937498 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38703 } } }
I20260810 21:09:52.937618 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:52.937670 5066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f23ca9b138647be9d7f8e8c8f0df703. Latest consensus state: current_term: 1 leader_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f23ca9b138647be9d7f8e8c8f0df703" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38703 } } }
I20260810 21:09:52.937803 5066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:52.937916 5078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:52.938419 5078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:52.938896 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 4945
I20260810 21:09:52.939031 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:52.939244 28117 external_mini_cluster.cc:1428] Setting key 22463af7f0466ef51f3cf6b2e150f06e
I20260810 21:09:52.940127 5078 catalog_manager.cc:1347] Generated new cluster ID: 313ca46430d24c4eb72bb8f7a741ed3f
I20260810 21:09:52.940184 5078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:52.945171 5078 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:52.945789 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.939677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.27.117.126:38703)
0810 21:09:52.939882 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.939890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.940309 (+ 419us) server_negotiation.cc:407] Connection header received
0810 21:09:52.940439 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.940447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.940514 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.940667 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:52.940867 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.942262 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.943318 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.943937 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.944078 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:52.944755 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:52.944759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:52.944759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:52.945140 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:52.945186 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:52.945202 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:52.945271 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:52.945397 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 21:09:52.945472 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260810 21:09:52.945815 5078 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:52.946899 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:09:52.952375 5078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f23ca9b138647be9d7f8e8c8f0df703: Generated new TSK 0
I20260810 21:09:52.952553 5078 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 21:09:52.983163 5105 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 21:09:52.983249 5105 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 21:09:52.983506 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:52.961151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.27.117.126:38703)
0810 21:09:52.961196 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:52.961197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:52.961313 (+ 116us) server_negotiation.cc:407] Connection header received
0810 21:09:52.961356 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:52.961357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:52.961370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:52.961438 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0810 21:09:52.961645 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.962667 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.963464 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.963684 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.963776 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:52.963966 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 21:09:52.983306 (+ 19340us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 21:09:52.983445 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 21:09:52.983575 5095 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 21:09:52.983628 5101 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:52.961055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38703 (local address 127.0.0.1:48290)
0810 21:09:52.961224 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:52.961235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:52.961302 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:52.961481 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:52.961482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:52.961484 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 21:09:52.961562 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:52.961564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.962766 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.962768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:52.963368 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:52.963370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:52.963787 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:52.963788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:52.963846 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:52.963852 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 21:09:52.983420 (+ 19568us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 21:09:52.983533 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.117.126:38703: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260810 21:09:52.984120 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 4945
2026-08-10T21:09:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 21:09:53.030046 28117 mini_oidc.cc:152] Starting JWKS server
I20260810 21:09:53.030731 28117 webserver.cc:460] Webserver started at https://127.0.0.1:37769/ using document root <none> and password file <none>
I20260810 21:09:53.030773 28117 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 21:09:53.030961 28117 webserver.cc:460] Webserver started at http://127.0.0.1:42789/ using document root <none> and password file <none>
2026-08-10T21:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:53Z Disabled control of system clock
I20260810 21:09:53.039589 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:36007
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:43341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:36007
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37769/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 21:09:53.120182 5114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:53.120402 5114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:53.120441 5114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:53.121595 5114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:53.121662 5114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:53.121683 5114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:53.121704 5114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:53.121735 5114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 21:09:53.121758 5114 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 21:09:53.121783 5114 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37769/jwks.json
I20260810 21:09:53.123123 5114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:36007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:36007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37769/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:53.123402 5114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:53.123684 5114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:53.128461 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:53.128475 5127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:53.128475 5121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:53.128851 5114 server_base.cc:1034] running on GCE node
I20260810 21:09:53.129101 5114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:53.129433 5114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:53.130743 5114 hybrid_clock.cc:648] HybridClock initialized: now 1786396193130723 us; error 29 us; skew 500 ppm
I20260810 21:09:53.131992 5114 webserver.cc:460] Webserver started at http://127.27.117.126:42203/ using document root <none> and password file <none>
I20260810 21:09:53.132237 5114 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:53.132321 5114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:53.132463 5114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:53.133679 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "0edce32787464e19883cdf9c7be3ae6b"
format_stamp: "Formatted at 2026-08-10 21:09:53 on dist-test-slave-2kcd"
server_key: "bdc06bcae1f5a8ef5b470fa53b6bebda"
server_key_iv: "ebd634b8dda1047094d8c8b2dc88fa58"
server_key_version: "encryptionkey@0"
I20260810 21:09:53.134071 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "0edce32787464e19883cdf9c7be3ae6b"
format_stamp: "Formatted at 2026-08-10 21:09:53 on dist-test-slave-2kcd"
server_key: "bdc06bcae1f5a8ef5b470fa53b6bebda"
server_key_iv: "ebd634b8dda1047094d8c8b2dc88fa58"
server_key_version: "encryptionkey@0"
I20260810 21:09:53.135486 5114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:09:53.136232 5135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:53.136415 5114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:53.136533 5114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "0edce32787464e19883cdf9c7be3ae6b"
format_stamp: "Formatted at 2026-08-10 21:09:53 on dist-test-slave-2kcd"
server_key: "bdc06bcae1f5a8ef5b470fa53b6bebda"
server_key_iv: "ebd634b8dda1047094d8c8b2dc88fa58"
server_key_version: "encryptionkey@0"
I20260810 21:09:53.136612 5114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:53.161036 5114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:53.161319 5114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:53.161479 5114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:53.165812 5114 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:36007
I20260810 21:09:53.165838 5234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:36007 every 8 connection(s)
I20260810 21:09:53.166179 5114 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:53.166872 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:53.169414 5236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b: Bootstrap starting.
I20260810 21:09:53.169997 5236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:53.170236 5236 log.cc:824] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:53.170919 5236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b: No bootstrap required, opened a new log
I20260810 21:09:53.172442 5236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edce32787464e19883cdf9c7be3ae6b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36007 } }
I20260810 21:09:53.172575 5236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:53.172597 5236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0edce32787464e19883cdf9c7be3ae6b, State: Initialized, Role: FOLLOWER
I20260810 21:09:53.172722 5236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edce32787464e19883cdf9c7be3ae6b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36007 } }
I20260810 21:09:53.172854 5236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:53.172901 5236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:53.172950 5236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:53.173467 5236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edce32787464e19883cdf9c7be3ae6b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36007 } }
I20260810 21:09:53.173593 5236 leader_election.cc:302] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0edce32787464e19883cdf9c7be3ae6b; no voters:
I20260810 21:09:53.173800 5236 leader_election.cc:288] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:53.173832 5241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:53.173885 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 5114
I20260810 21:09:53.173941 5241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [term 1 LEADER]: Becoming Leader. State: Replica: 0edce32787464e19883cdf9c7be3ae6b, State: Running, Role: LEADER
I20260810 21:09:53.173992 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:53.174054 5236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:53.174187 28117 external_mini_cluster.cc:1428] Setting key 97ea41e0cbdf82c5716d258f1141c1f0
I20260810 21:09:53.174031 5241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edce32787464e19883cdf9c7be3ae6b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36007 } }
I20260810 21:09:53.174774 5242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0edce32787464e19883cdf9c7be3ae6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edce32787464e19883cdf9c7be3ae6b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36007 } } }
I20260810 21:09:53.174866 5242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:53.175088 5259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:53.175621 5259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:53.176059 5243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0edce32787464e19883cdf9c7be3ae6b. Latest consensus state: current_term: 1 leader_uuid: "0edce32787464e19883cdf9c7be3ae6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0edce32787464e19883cdf9c7be3ae6b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36007 } } }
I20260810 21:09:53.176185 5243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:53.179684 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:53.174615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.27.117.126:36007)
0810 21:09:53.174789 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:53.174798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:53.174846 (+ 48us) server_negotiation.cc:407] Connection header received
0810 21:09:53.175112 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:53.175122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:53.175207 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:53.175344 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:53.175611 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.177036 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.177855 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.178256 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.178387 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:53.179056 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:53.179062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:53.179066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:53.179243 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:53.179275 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:53.179284 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:53.179364 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:53.179437 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 21:09:53.179513 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260810 21:09:53.180667 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:09:53.185482 5259 catalog_manager.cc:1347] Generated new cluster ID: 2c78c1213942408fbaf8404d8cc990da
I20260810 21:09:53.185539 5259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:53.205235 5259 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:53.205938 5259 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:53.212430 5259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0edce32787464e19883cdf9c7be3ae6b: Generated new TSK 0
I20260810 21:09:53.212599 5259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:09:53.229326 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:53.225574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.27.117.126:36007)
0810 21:09:53.225641 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:53.225644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:53.225790 (+ 146us) server_negotiation.cc:407] Connection header received
0810 21:09:53.225857 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:53.225858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:53.225871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:53.225936 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0810 21:09:53.226092 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.226593 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.227742 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.227943 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.228002 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:53.228194 (+ 192us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 21:09:53.229068 (+ 874us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 21:09:53.229242 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 21:09:53.229385 5247 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 21:09:53.229391 5293 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:53.225520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36007 (local address 127.0.0.1:41440)
0810 21:09:53.225712 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:53.225724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:53.225793 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:53.225954 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:53.225955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:53.225957 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 21:09:53.226023 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:53.226025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.226689 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.226690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:53.227654 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:53.227657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.228032 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.228033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:53.228095 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:53.228101 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 21:09:53.229190 (+ 1089us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 21:09:53.229286 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.117.126:36007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37769/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260810 21:09:53.229717 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 5114
E20260810 21:09:53.255287 5296 webserver.cc:533] Webserver: sslize failed:
E20260810 21:09:53.255368 5296 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T21:09:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (253 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T21:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:53Z Disabled control of system clock
I20260810 21:09:53.291998 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:33987
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:33987
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/testchainca.cert with env {}
W20260810 21:09:53.372757 5302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:53.373011 5302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:53.373049 5302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:53.374191 5302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:53.374264 5302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:53.374285 5302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:53.374306 5302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:53.374336 5302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 21:09:53.374359 5302 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 21:09:53.374387 5302 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 21:09:53.375936 5302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:33987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:33987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:53.376211 5302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:53.376466 5302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:53.381283 5312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:53.381304 5310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:53.381524 5309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:53.381523 5302 server_base.cc:1034] running on GCE node
I20260810 21:09:53.381852 5302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:53.382175 5302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:53.383344 5302 hybrid_clock.cc:648] HybridClock initialized: now 1786396193383320 us; error 37 us; skew 500 ppm
I20260810 21:09:53.384591 5302 webserver.cc:460] Webserver started at http://127.27.117.126:40733/ using document root <none> and password file <none>
I20260810 21:09:53.384922 5302 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:53.384997 5302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:53.385134 5302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:53.386307 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "ffef5a78e2434a3f9d60ea838b2cb813"
format_stamp: "Formatted at 2026-08-10 21:09:53 on dist-test-slave-2kcd"
server_key: "52d41a9dbcfc851ddb3acc9e9a662a20"
server_key_iv: "1dc895df25b4119a2494f3b0372570dd"
server_key_version: "encryptionkey@0"
I20260810 21:09:53.386698 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "ffef5a78e2434a3f9d60ea838b2cb813"
format_stamp: "Formatted at 2026-08-10 21:09:53 on dist-test-slave-2kcd"
server_key: "52d41a9dbcfc851ddb3acc9e9a662a20"
server_key_iv: "1dc895df25b4119a2494f3b0372570dd"
server_key_version: "encryptionkey@0"
I20260810 21:09:53.388073 5302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:53.388845 5322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:53.389038 5302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:53.389139 5302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "ffef5a78e2434a3f9d60ea838b2cb813"
format_stamp: "Formatted at 2026-08-10 21:09:53 on dist-test-slave-2kcd"
server_key: "52d41a9dbcfc851ddb3acc9e9a662a20"
server_key_iv: "1dc895df25b4119a2494f3b0372570dd"
server_key_version: "encryptionkey@0"
I20260810 21:09:53.389221 5302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:53.398644 5302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:53.398926 5302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:53.399071 5302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:53.402839 5302 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:33987
I20260810 21:09:53.402905 5413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:33987 every 8 connection(s)
I20260810 21:09:53.403235 5302 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:53.404217 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:53.406391 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 5302
I20260810 21:09:53.406507 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:53.406561 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813: Bootstrap starting.
I20260810 21:09:53.406719 28117 external_mini_cluster.cc:1428] Setting key 78fe30b796d6af37f110e6b4b04c000a
I20260810 21:09:53.407331 5414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:53.407665 5414 log.cc:824] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:53.408351 5414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813: No bootstrap required, opened a new log
I20260810 21:09:53.410171 5414 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33987 } }
I20260810 21:09:53.410300 5414 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:53.410323 5414 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffef5a78e2434a3f9d60ea838b2cb813, State: Initialized, Role: FOLLOWER
I20260810 21:09:53.410478 5414 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33987 } }
I20260810 21:09:53.410557 5414 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:53.410580 5414 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:53.410650 5414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:53.411237 5414 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33987 } }
I20260810 21:09:53.411352 5414 leader_election.cc:302] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffef5a78e2434a3f9d60ea838b2cb813; no voters:
I20260810 21:09:53.411554 5414 leader_election.cc:288] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:53.411746 5425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:53.411825 5414 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:53.411897 5425 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [term 1 LEADER]: Becoming Leader. State: Replica: ffef5a78e2434a3f9d60ea838b2cb813, State: Running, Role: LEADER
I20260810 21:09:53.411989 5425 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33987 } }
I20260810 21:09:53.412300 5425 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffef5a78e2434a3f9d60ea838b2cb813. Latest consensus state: current_term: 1 leader_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33987 } } }
I20260810 21:09:53.412374 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:53.407218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.27.117.126:33987)
0810 21:09:53.407453 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:53.407462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:53.407486 (+ 24us) server_negotiation.cc:407] Connection header received
0810 21:09:53.407952 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:53.407968 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:53.408023 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:53.408170 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:53.408353 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.409385 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.410169 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.410758 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.410913 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:53.411572 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:53.411576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:53.411576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:09:53.411790 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:53.411830 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:09:53.411845 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:53.411916 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:53.412064 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 21:09:53.412138 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260810 21:09:53.412384 5425 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:53.412532 5424 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffef5a78e2434a3f9d60ea838b2cb813" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33987 } } }
I20260810 21:09:53.412618 5424 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:53.412738 5437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:53.413333 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:09:53.413338 5437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:53.415223 5437 catalog_manager.cc:1347] Generated new cluster ID: 465be5fdc4c44545aaf834ec65adfc6d
I20260810 21:09:53.415294 5437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:09:53.423555 5437 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:53.424042 5437 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:53.429492 5437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffef5a78e2434a3f9d60ea838b2cb813: Generated new TSK 0
I20260810 21:09:53.429663 5437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:09:53.534282 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:53.435844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.27.117.126:33987)
0810 21:09:53.435908 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:53.435910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:53.436033 (+ 123us) server_negotiation.cc:407] Connection header received
0810 21:09:53.436071 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:53.436072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:53.436083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:53.436163 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0810 21:09:53.436276 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.437343 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.438125 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.438438 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.438511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:53.438655 (+ 144us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 21:09:53.533996 (+ 95341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 21:09:53.534189 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37218: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 21:09:53.534369 5422 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37218: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 21:09:53.534396 5462 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:53.435781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:33987 (local address 127.0.0.1:37218)
0810 21:09:53.435954 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:53.435964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:53.436024 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:53.436166 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:53.436166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:53.436168 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 21:09:53.436215 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:53.436217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.437467 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.437468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:53.438045 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:53.438046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:53.438521 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:53.438522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:53.438560 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:53.438564 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 21:09:53.534156 (+ 95592us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 21:09:53.534280 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.117.126:33987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260810 21:09:53.534795 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 5302
2026-08-10T21:09:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (277 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:53 dist-test-slave-2kcd krb5kdc[5486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:53 dist-test-slave-2kcd krb5kdc[5486](info): set up 2 sockets
Aug 10 21:09:53 dist-test-slave-2kcd krb5kdc[5486](info): commencing operation
krb5kdc: starting...
W20260810 21:09:55.012768 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:09:55.129240 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:43621
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:55.211551 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:55.211747 5504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:55.211769 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:55.212967 5504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:55.213009 5504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:55.213023 5504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:55.213043 5504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:55.213095 5504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:55.214586 5504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:55.214775 5504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:55.215018 5504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:55.219849 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.219849 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.219857 5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:55.219949 5504 server_base.cc:1034] running on GCE node
I20260810 21:09:55.220350 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:55.220665 5504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:55.221832 5504 hybrid_clock.cc:648] HybridClock initialized: now 1786396195221824 us; error 37 us; skew 500 ppm
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.224359 5504 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:09:55.224867 5504 webserver.cc:460] Webserver started at http://127.27.117.126:34343/ using document root <none> and password file <none>
I20260810 21:09:55.225080 5504 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:55.225121 5504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:55.225206 5504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:55.226186 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "8ea0548fca86f1326ab1c51b3f01c3be"
server_key_iv: "df678d706c1ec8855323903e977edbdf"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.226475 5504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "8ea0548fca86f1326ab1c51b3f01c3be"
server_key_iv: "df678d706c1ec8855323903e977edbdf"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.227794 5504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:55.228463 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.228637 5504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:55.228731 5504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "8ea0548fca86f1326ab1c51b3f01c3be"
server_key_iv: "df678d706c1ec8855323903e977edbdf"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.228839 5504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:55.239595 5504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:55.241597 5504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:55.241755 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:55.245838 5504 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:43621
I20260810 21:09:55.245891 5617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:43621 every 8 connection(s)
I20260810 21:09:55.246244 5504 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:55.247110 5618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:55.249606 5618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3: Bootstrap starting.
I20260810 21:09:55.250203 5618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:55.250505 5618 log.cc:824] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:55.251188 5618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3: No bootstrap required, opened a new log
I20260810 21:09:55.252854 5618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43621 } }
I20260810 21:09:55.253022 5618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:55.253082 5618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3, State: Initialized, Role: FOLLOWER
I20260810 21:09:55.253254 5618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43621 } }
I20260810 21:09:55.253357 5618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:55.253396 5618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:55.253453 5618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:55.253801 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 5504
I20260810 21:09:55.253881 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:55.254020 5618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43621 } }
I20260810 21:09:55.254082 28117 external_mini_cluster.cc:1428] Setting key a48a7ea5e0acdb18409bef31152be994
I20260810 21:09:55.254144 5618 leader_election.cc:302] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3; no voters:
I20260810 21:09:55.254333 5618 leader_election.cc:288] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:55.254369 5626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:55.254721 5618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:55.254786 5626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [term 1 LEADER]: Becoming Leader. State: Replica: 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3, State: Running, Role: LEADER
I20260810 21:09:55.254914 5626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43621 } }
I20260810 21:09:55.255282 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43621 } } }
I20260810 21:09:55.255374 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:55.255501 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3. Latest consensus state: current_term: 1 leader_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43621 } } }
I20260810 21:09:55.255570 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:55.255966 5642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:55.256505 5642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:55.258476 5642 catalog_manager.cc:1347] Generated new cluster ID: 5e4c476c14294e0d849de748fe9291da
I20260810 21:09:55.258527 5642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:55.266911 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.254599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.27.117.126:43621)
0810 21:09:55.255130 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.255139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.255172 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:09:55.255291 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.255297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.255354 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.256475 (+ 1121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.256571 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.258027 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.259126 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.259725 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.259842 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.262687 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.262691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.262692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.262703 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.265302 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.265710 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.265713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.265713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.265814 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.266028 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.266029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.266029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.266313 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.266463 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.266572 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.266699 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":112,"threads_started":1}
I20260810 21:09:55.271025 5642 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:55.271518 5642 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:55.279673 5642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48e5e0d9ea1f4c80bd3bb7ed1fdfe9c3: Generated new TSK 0
I20260810 21:09:55.279844 5642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:09:55.309610 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:43621
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:55.417806 5672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:55.417994 5672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:55.418017 5672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:55.419143 5672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:55.419184 5672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:55.419210 5672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:55.420665 5672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:55.420934 5672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:55.421156 5672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:55.426016 5678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.425951 5677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:55.426088 5672 server_base.cc:1034] running on GCE node
W20260810 21:09:55.425949 5681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:55.426440 5672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:55.426651 5672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:55.427791 5672 hybrid_clock.cc:648] HybridClock initialized: now 1786396195427780 us; error 30 us; skew 500 ppm
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.430305 5672 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:09:55.430721 5672 webserver.cc:460] Webserver started at http://127.27.117.65:34133/ using document root <none> and password file <none>
I20260810 21:09:55.430965 5672 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:55.431010 5672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:55.431171 5672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:55.432216 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "36300f6a0b7c4780bf54429897229b89"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "3615aeae9ea7e3dd931ab45e44ad6244"
server_key_iv: "41ee842470c63b1160406e51f3eba563"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.432680 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "36300f6a0b7c4780bf54429897229b89"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "3615aeae9ea7e3dd931ab45e44ad6244"
server_key_iv: "41ee842470c63b1160406e51f3eba563"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.434068 5672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:55.434767 5695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.434930 5672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.435031 5672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "36300f6a0b7c4780bf54429897229b89"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "3615aeae9ea7e3dd931ab45e44ad6244"
server_key_iv: "41ee842470c63b1160406e51f3eba563"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.435102 5672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:55.449361 5672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:55.451089 5672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:55.451248 5672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:55.451471 5672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:55.451781 5672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:55.451834 5672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.451870 5672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:55.451896 5672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.458175 5672 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:39361
I20260810 21:09:55.458213 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:39361 every 8 connection(s)
I20260810 21:09:55.458524 5672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:55.464893 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 5672
I20260810 21:09:55.465013 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:55.465212 28117 external_mini_cluster.cc:1428] Setting key 1c3f8484b48dc9f7b9309e746e87486e
I20260810 21:09:55.468001 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.459246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:43621 (local address 127.27.117.65:40679)
0810 21:09:55.459428 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:55.459468 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:55.460363 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:55.460571 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:55.460574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:55.460832 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:55.461015 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:55.461019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.461668 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.461669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:55.462424 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:55.462426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.462751 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.462752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:55.462840 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.463271 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:55.463279 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:55.464705 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:55.466715 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:55.466721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:55.466722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:55.466932 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:55.467134 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:55.467134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:55.467134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:55.467229 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:55.467491 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:55.467495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:55.467603 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 21:09:55.467731 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 21:09:55.467849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260810 21:09:55.468003 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.459288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:40679 (local address 127.27.117.126:43621)
0810 21:09:55.459349 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.459351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.459560 (+ 209us) server_negotiation.cc:407] Connection header received
0810 21:09:55.460450 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.460451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.460465 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.460521 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.461089 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.461586 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.462495 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.462682 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.462731 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.464846 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.464847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.464847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.464854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.466631 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.467005 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.467007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.467007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.467069 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.467290 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.467291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.467291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.467429 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.467497 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.467770 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.467870 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:09:55.468497 5887 heartbeater.cc:347] Connected to a master server at 127.27.117.126:43621
I20260810 21:09:55.468601 5887 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:55.468822 5887 heartbeater.cc:511] Master 127.27.117.126:43621 requested a full tablet report, sending...
I20260810 21:09:55.469363 5560 ts_manager.cc:194] Registered new tserver with Master: 36300f6a0b7c4780bf54429897229b89 (127.27.117.65:39361)
I20260810 21:09:55.469978 5560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:40679
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:09:55.490809 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:43621
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:55.571703 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:55.571926 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:55.571964 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:55.573156 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:55.573215 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:55.573261 5897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:55.574746 5897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:55.575028 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:55.575254 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:55.580214 5906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.580227 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.580248 5905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:55.580474 5897 server_base.cc:1034] running on GCE node
I20260810 21:09:55.580675 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:55.580943 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:55.582114 5897 hybrid_clock.cc:648] HybridClock initialized: now 1786396195582102 us; error 29 us; skew 500 ppm
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.584640 5897 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:09:55.585104 5897 webserver.cc:460] Webserver started at http://127.27.117.66:33173/ using document root <none> and password file <none>
I20260810 21:09:55.585345 5897 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:55.585390 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:55.585559 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:55.586568 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "63baf23010524018b337480f2046896d"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "3f6e7fbbcfa1524dd8f6455ece5c54e9"
server_key_iv: "bd66271b9664dc8ad301cbb913c08192"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.586925 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "63baf23010524018b337480f2046896d"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "3f6e7fbbcfa1524dd8f6455ece5c54e9"
server_key_iv: "bd66271b9664dc8ad301cbb913c08192"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.588230 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:55.589010 5928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.589181 5897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.589290 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "63baf23010524018b337480f2046896d"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "3f6e7fbbcfa1524dd8f6455ece5c54e9"
server_key_iv: "bd66271b9664dc8ad301cbb913c08192"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.589365 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:55.603399 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:55.605722 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:55.605885 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:55.606113 5897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:55.606420 5897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:55.606474 5897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.606509 5897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:55.606539 5897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.612355 5897 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36115
I20260810 21:09:55.612391 6119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36115 every 8 connection(s)
I20260810 21:09:55.612710 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:55.615746 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 5897
I20260810 21:09:55.615851 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:55.616034 28117 external_mini_cluster.cc:1428] Setting key 15445591e58b7867f2dc6f74e4767ec3
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:55.623126 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.613451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:60797 (local address 127.27.117.126:43621)
0810 21:09:55.613528 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.613531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.613717 (+ 186us) server_negotiation.cc:407] Connection header received
0810 21:09:55.614670 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.614671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.614684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.614748 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.615337 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.616124 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.617350 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.617665 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.617741 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.620137 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.620138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.620138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.620144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.621533 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.622053 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.622054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.622054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.622105 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.622414 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.622415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.622416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.622555 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.622628 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.622968 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.623047 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:09:55.623409 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.613408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:43621 (local address 127.27.117.66:60797)
0810 21:09:55.613594 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:55.613634 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:55.614558 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:55.614816 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:55.614820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:55.615063 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:55.615237 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:55.615241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.616240 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.616242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:55.617260 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:55.617263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.617754 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.617755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:55.617858 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.618342 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:55.618350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:55.620037 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:55.621621 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:55.621629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:55.621631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:55.621970 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:55.622174 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:55.622175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:55.622175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:55.622332 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:55.622635 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:55.622642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:55.622816 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 21:09:55.622947 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 21:09:55.623142 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260810 21:09:55.623952 6120 heartbeater.cc:347] Connected to a master server at 127.27.117.126:43621
I20260810 21:09:55.624059 6120 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:55.624245 6120 heartbeater.cc:511] Master 127.27.117.126:43621 requested a full tablet report, sending...
I20260810 21:09:55.624652 5560 ts_manager.cc:194] Registered new tserver with Master: 63baf23010524018b337480f2046896d (127.27.117.66:36115)
I20260810 21:09:55.625105 5560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:60797
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:09:55.641659 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:43621
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:55.723963 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:55.724225 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:55.724263 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:55.725466 6132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:55.725535 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:55.725582 6132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:55.727100 6132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:55.727391 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:55.727617 6132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:55.732579 6142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.732609 6144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:55.732667 6140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:55.733237 6132 server_base.cc:1034] running on GCE node
I20260810 21:09:55.733464 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:55.733743 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:55.734913 6132 hybrid_clock.cc:648] HybridClock initialized: now 1786396195734888 us; error 37 us; skew 500 ppm
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.737560 6132 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:09:55.738129 6132 webserver.cc:460] Webserver started at http://127.27.117.67:43155/ using document root <none> and password file <none>
I20260810 21:09:55.738366 6132 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:55.738435 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:55.738583 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:55.739621 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "e490736849fd45dabf72ae9d56c6c1c9"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "d293739115b6adaa8e0a3e20e5c5d762"
server_key_iv: "771d3c61da1b1817c9c2eae24f058fba"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.739986 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "e490736849fd45dabf72ae9d56c6c1c9"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "d293739115b6adaa8e0a3e20e5c5d762"
server_key_iv: "771d3c61da1b1817c9c2eae24f058fba"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.741423 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:55.742187 6157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.742357 6132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:55.742465 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "e490736849fd45dabf72ae9d56c6c1c9"
format_stamp: "Formatted at 2026-08-10 21:09:55 on dist-test-slave-2kcd"
server_key: "d293739115b6adaa8e0a3e20e5c5d762"
server_key_iv: "771d3c61da1b1817c9c2eae24f058fba"
server_key_version: "encryptionkey@0"
I20260810 21:09:55.742538 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:55.759888 6132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:55.762231 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:55.762392 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:55.762622 6132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:55.762923 6132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:55.762979 6132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.763020 6132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:55.763075 6132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:55.769404 6132 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:43913
I20260810 21:09:55.769464 6347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:43913 every 8 connection(s)
I20260810 21:09:55.769789 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:09:55.776520 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 6132
I20260810 21:09:55.776618 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:55.776857 28117 external_mini_cluster.cc:1428] Setting key f8b959bb3f9c8780a420140acfeffd48
I20260810 21:09:55.779008 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.770482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:52895 (local address 127.27.117.126:43621)
0810 21:09:55.770556 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.770558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.770725 (+ 167us) server_negotiation.cc:407] Connection header received
0810 21:09:55.771597 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.771598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.771608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.771659 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.772196 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.772684 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.773605 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.773789 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.773844 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.775874 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.775876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.775876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.775882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.777522 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.778046 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.778047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.778047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.778112 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.778371 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.778372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.778372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.778489 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.778560 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.778840 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.778918 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 21:09:55.779196 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.770440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:43621 (local address 127.27.117.67:52895)
0810 21:09:55.770607 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:55.770642 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:55.771512 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:55.771710 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:55.771713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:55.771938 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:55.772129 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:55.772133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.772760 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.772760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:55.773534 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:55.773535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.773863 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.773863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:55.773956 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.774399 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:55.774407 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:55.775773 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:55.777611 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:55.777618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:55.777620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:55.777947 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:55.778178 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:55.778179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:55.778180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:55.778295 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:55.778573 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:55.778580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:55.778708 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 21:09:55.778815 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 21:09:55.778967 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260810 21:09:55.779630 6349 heartbeater.cc:347] Connected to a master server at 127.27.117.126:43621
I20260810 21:09:55.779727 6349 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:55.779906 6349 heartbeater.cc:511] Master 127.27.117.126:43621 requested a full tablet report, sending...
I20260810 21:09:55.780272 5560 ts_manager.cc:194] Registered new tserver with Master: e490736849fd45dabf72ae9d56c6c1c9 (127.27.117.67:43913)
I20260810 21:09:55.780646 5560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:52895
I20260810 21:09:55.788636 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/test-admin.keytab.
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.804214 28117 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 21:09:55.805927 6356 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:55.805053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.27.117.65:39361)
0810 21:09:55.805212 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.805216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.805227 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:55.805270 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.805272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.805285 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.805351 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.805841 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46508: BlockingRecv error: recv got EOF from 127.0.0.1:46508 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260810 21:09:55.806063 28117 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc' of type FILE
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.806913 28117 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.807765 28117 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:55.815486 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.808270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46524 (local address 127.27.117.65:39361)
0810 21:09:55.808334 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.808336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.808377 (+ 41us) server_negotiation.cc:407] Connection header received
0810 21:09:55.808441 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.808442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.808454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.808526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.809010 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.809959 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.810694 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.811037 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.811119 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.813136 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.813144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.813145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.813151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.814547 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.814860 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.814862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.814863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.814913 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.815090 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.815090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.815090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.815203 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.815291 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.815361 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.815416 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 21:09:55.817308 6356 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:55.816690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.27.117.65:39361)
0810 21:09:55.816727 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.816729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.816756 (+ 27us) server_negotiation.cc:407] Connection header received
0810 21:09:55.816915 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.816915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.816924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.816969 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.817268 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46526: BlockingRecv error: recv got EOF from 127.0.0.1:46526 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 21:09:55.817431 28117 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc' of type FILE
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.818313 28117 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.819123 28117 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:55.825644 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.819607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.27.117.65:39361)
0810 21:09:55.819643 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.819644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.819698 (+ 54us) server_negotiation.cc:407] Connection header received
0810 21:09:55.819745 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.819746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.819754 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.819796 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.820111 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.820565 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.821238 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.821426 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.821477 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.823321 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.823323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.823323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.823329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.824677 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.825057 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.825058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.825058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.825108 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.825253 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.825254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.825254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.825327 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.825388 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.825470 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.825524 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 21:09:55.827093 6356 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:09:55.826526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.27.117.65:39361)
0810 21:09:55.826582 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.826584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.826632 (+ 48us) server_negotiation.cc:407] Connection header received
0810 21:09:55.826702 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.826703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.826712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.826781 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.827053 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46548: BlockingRecv error: recv got EOF from 127.0.0.1:46548 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 21:09:55.827201 28117 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786396166878326-28117-0/krb5kdc/krb5cc' of type FILE
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.828092 28117 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:55.829023 28117 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[5486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:09:55.835572 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:55.829446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.27.117.65:39361)
0810 21:09:55.829495 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:55.829496 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:55.829505 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:55.829525 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:55.829526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:55.829544 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:55.829587 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:55.829885 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.830297 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.830938 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:55.831106 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:55.831155 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:55.833413 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:55.833415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:55.833415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:55.833422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:55.834479 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.834827 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.834829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.834829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.834882 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:55.835105 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:55.835106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:55.835106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:55.835203 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:55.835308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:55.835407 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 21:09:55.835479 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 21:09:55.836141 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 5672
I20260810 21:09:55.851338 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 5897
I20260810 21:09:55.868675 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 6132
I20260810 21:09:55.880714 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 5504
2026-08-10T21:09:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2166 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[6378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[6378](info): set up 2 sockets
Aug 10 21:09:55 dist-test-slave-2kcd krb5kdc[6378](info): commencing operation
krb5kdc: starting...
W20260810 21:09:57.143229 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:09:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab.
Entry for principal oryx/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab.
I20260810 21:09:57.257594 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38643
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab
--principal=oryx/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:57.337901 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:57.338132 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:57.338174 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:57.339315 6403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:57.339382 6403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:57.339406 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:57.339434 6403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:57.339464 6403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:57.340930 6403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab
--principal=oryx/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:57.341187 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:57.341405 6403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:57.346501 6416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.346619 6403 server_base.cc:1034] running on GCE node
W20260810 21:09:57.346531 6415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:57.346520 6421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.346982 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:57.347368 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:57.348531 6403 hybrid_clock.cc:648] HybridClock initialized: now 1786396197348518 us; error 44 us; skew 500 ppm
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:57.353654 6403 init.cc:377] Logged in from keytab as oryx/127.27.117.126@KRBTEST.COM (short username oryx)
I20260810 21:09:57.354082 6403 webserver.cc:460] Webserver started at http://127.27.117.126:41155/ using document root <none> and password file <none>
I20260810 21:09:57.354326 6403 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:57.354394 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:57.354553 6403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:57.355628 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a7587b80d5004d33880aa2b91613e611"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "d9a365d6d57ebbffe9e1755f095aa95b"
server_key_iv: "6cdddc9039eaa31facd1dba5561b12bf"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.356015 6403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a7587b80d5004d33880aa2b91613e611"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "d9a365d6d57ebbffe9e1755f095aa95b"
server_key_iv: "6cdddc9039eaa31facd1dba5561b12bf"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.357432 6403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:57.358124 6431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.358315 6403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:57.358413 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a7587b80d5004d33880aa2b91613e611"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "d9a365d6d57ebbffe9e1755f095aa95b"
server_key_iv: "6cdddc9039eaa31facd1dba5561b12bf"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.358494 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:57.390100 6403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:57.391983 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:57.392149 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:57.396507 6403 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:38643
I20260810 21:09:57.396589 6532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:38643 every 8 connection(s)
I20260810 21:09:57.396952 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:09:57.397936 6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:57.400300 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Bootstrap starting.
I20260810 21:09:57.400957 6533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:57.401257 6533 log.cc:824] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:57.401942 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: No bootstrap required, opened a new log
I20260810 21:09:57.402181 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 6403
I20260810 21:09:57.402269 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:09:57.402494 28117 external_mini_cluster.cc:1428] Setting key f3894ffcff5491d5c3cb5f7523708371
I20260810 21:09:57.404258 6533 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } }
I20260810 21:09:57.404436 6533 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:57.404477 6533 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7587b80d5004d33880aa2b91613e611, State: Initialized, Role: FOLLOWER
I20260810 21:09:57.404649 6533 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } }
I20260810 21:09:57.404718 6533 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:09:57.404742 6533 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:09:57.404825 6533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:57.405866 6533 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } }
I20260810 21:09:57.405975 6533 leader_election.cc:302] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7587b80d5004d33880aa2b91613e611; no voters:
I20260810 21:09:57.406124 6533 leader_election.cc:288] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:09:57.406174 6547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:57.406368 6547 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 LEADER]: Becoming Leader. State: Replica: a7587b80d5004d33880aa2b91613e611, State: Running, Role: LEADER
I20260810 21:09:57.406368 6533 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:57.406569 6547 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } }
I20260810 21:09:57.408075 6549 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } }
I20260810 21:09:57.408191 6550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7587b80d5004d33880aa2b91613e611. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } }
I20260810 21:09:57.408192 6549 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:57.408272 6550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
W20260810 21:09:57.408826 6577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:09:57.408880 6577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:09:57.408921 6578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:09:57.409078 6578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:09:57.410686 6578 catalog_manager.cc:1347] Generated new cluster ID: e3f5693b3d654f8498d63913b10dcb90
I20260810 21:09:57.410739 6578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:09:57.414561 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.403041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.27.117.126:38643)
0810 21:09:57.403254 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:57.403264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:57.403302 (+ 38us) server_negotiation.cc:407] Connection header received
0810 21:09:57.404221 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:57.404230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:57.404294 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:57.404446 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:57.405265 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.406707 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.407750 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.408432 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.408571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.411272 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:57.411276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:57.411276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:57.411287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:57.413076 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.413421 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.413424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.413424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.413538 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.413751 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.413752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.413752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.413992 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:57.414200 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:57.414268 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 21:09:57.414408 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260810 21:09:57.435452 6578 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:09:57.436051 6578 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:09:57.440603 6578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Generated new TSK 0
I20260810 21:09:57.440809 6578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab.
Entry for principal oryx/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab.
I20260810 21:09:57.472478 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38643
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:57.554461 6587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:57.554662 6587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:57.554685 6587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:57.555760 6587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:57.555794 6587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:57.555819 6587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:57.557201 6587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:57.557375 6587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:57.557564 6587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:57.562353 6601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:57.562407 6597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.562470 6587 server_base.cc:1034] running on GCE node
W20260810 21:09:57.562600 6595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.562834 6587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:57.563027 6587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:57.564165 6587 hybrid_clock.cc:648] HybridClock initialized: now 1786396197564155 us; error 28 us; skew 500 ppm
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:57.566708 6587 init.cc:377] Logged in from keytab as oryx/127.27.117.65@KRBTEST.COM (short username oryx)
I20260810 21:09:57.567164 6587 webserver.cc:460] Webserver started at http://127.27.117.65:41839/ using document root <none> and password file <none>
I20260810 21:09:57.567386 6587 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:57.567453 6587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:57.567611 6587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:57.568753 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "48ecce09cf3248f2bddc714750d40946"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "77d329bac7aab88c561271bd62bffa9f"
server_key_iv: "55cecd0b7a37aed10490693617a34709"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.569199 6587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "48ecce09cf3248f2bddc714750d40946"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "77d329bac7aab88c561271bd62bffa9f"
server_key_iv: "55cecd0b7a37aed10490693617a34709"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.570521 6587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:57.571226 6611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.571413 6587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:57.571516 6587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "48ecce09cf3248f2bddc714750d40946"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "77d329bac7aab88c561271bd62bffa9f"
server_key_iv: "55cecd0b7a37aed10490693617a34709"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.571611 6587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:57.584730 6587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:57.586951 6587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:57.587117 6587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:57.587350 6587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:57.587669 6587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:57.587725 6587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.587759 6587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:57.587787 6587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.593657 6587 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:36283
I20260810 21:09:57.593703 6816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:36283 every 8 connection(s)
I20260810 21:09:57.594009 6587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:57.597683 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 6587
I20260810 21:09:57.597798 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:09:57.597983 28117 external_mini_cluster.cc:1428] Setting key 5df90390ed8092a67c385b974895d0b5
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:09:57.604097 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.594757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:34085)
0810 21:09:57.594968 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:57.595001 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:57.595508 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:57.595764 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:57.595770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:57.596338 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:57.596518 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:57.596522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.597305 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.597306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:57.598279 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:57.598282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.598681 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.598682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:57.598843 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.599320 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:57.599330 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:57.600984 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:57.602799 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:57.602805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:57.602807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:57.603020 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:57.603248 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:57.603248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:57.603248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:57.603343 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:57.603640 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:57.603645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:57.603752 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 21:09:57.603829 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 21:09:57.603943 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260810 21:09:57.604102 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.594786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:34085 (local address 127.27.117.126:38643)
0810 21:09:57.594830 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:57.594832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:57.595087 (+ 255us) server_negotiation.cc:407] Connection header received
0810 21:09:57.595623 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:57.595624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:57.595640 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:57.595688 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:57.596615 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.597180 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.598348 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.598570 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.598657 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.601084 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:57.601086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:57.601086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:57.601098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:57.602721 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.603096 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.603097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.603098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.603173 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.603410 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.603411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.603412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.603565 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:57.603639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:57.603886 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 21:09:57.603982 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:09:57.604590 6817 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38643
I20260810 21:09:57.604702 6817 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:57.604920 6817 heartbeater.cc:511] Master 127.27.117.126:38643 requested a full tablet report, sending...
I20260810 21:09:57.605459 6465 ts_manager.cc:194] Registered new tserver with Master: 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:09:57.606125 6465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:34085
WARNING: no policy specified for oryx/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab.
Entry for principal oryx/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab.
I20260810 21:09:57.623440 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38643
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:57.703815 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:57.704043 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:57.704082 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:57.705242 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:57.705312 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:57.705356 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:57.706955 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:57.707221 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:57.707464 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:57.712635 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.712700 6824 server_base.cc:1034] running on GCE node
W20260810 21:09:57.712637 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:57.712632 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.713076 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:57.713393 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:57.714542 6824 hybrid_clock.cc:648] HybridClock initialized: now 1786396197714526 us; error 40 us; skew 500 ppm
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:57.717087 6824 init.cc:377] Logged in from keytab as oryx/127.27.117.66@KRBTEST.COM (short username oryx)
I20260810 21:09:57.717589 6824 webserver.cc:460] Webserver started at http://127.27.117.66:44857/ using document root <none> and password file <none>
I20260810 21:09:57.717839 6824 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:57.717908 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:57.718055 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:57.719054 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "ff0643f61c584a53a0eda4c92231ba22"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "8f153efcedaad54cec2287ea17710b10"
server_key_iv: "f4cd38ce243e8db5dabe779ce39c5dc3"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.719417 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "ff0643f61c584a53a0eda4c92231ba22"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "8f153efcedaad54cec2287ea17710b10"
server_key_iv: "f4cd38ce243e8db5dabe779ce39c5dc3"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.720716 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:09:57.721473 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.721642 6824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:57.721747 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "ff0643f61c584a53a0eda4c92231ba22"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "8f153efcedaad54cec2287ea17710b10"
server_key_iv: "f4cd38ce243e8db5dabe779ce39c5dc3"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.721825 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:57.732054 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:57.733646 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:57.733789 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:57.734004 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:57.734284 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:57.734346 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.734378 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:57.734404 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.740257 6824 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36243
I20260810 21:09:57.740301 7048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36243 every 8 connection(s)
I20260810 21:09:57.740579 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:09:57.747925 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 6824
I20260810 21:09:57.748015 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:09:57.748202 28117 external_mini_cluster.cc:1428] Setting key a53f14d6c780ff66c608adc03d5b213a
I20260810 21:09:57.749647 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.741318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:47231 (local address 127.27.117.126:38643)
0810 21:09:57.741365 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:57.741367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:57.741590 (+ 223us) server_negotiation.cc:407] Connection header received
0810 21:09:57.742097 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:57.742099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:57.742113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:57.742161 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:57.743074 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.743584 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.744529 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.744739 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.744839 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.746950 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:57.746951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:57.746952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:57.746959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:57.748248 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.748651 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.748652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.748653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.748718 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.748996 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.748997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.748998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.749124 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:57.749204 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:57.749472 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 21:09:57.749559 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:09:57.749835 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.741258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:47231)
0810 21:09:57.741458 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:57.741496 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:57.741984 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:57.742235 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:57.742238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:57.742789 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:57.742962 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:57.742966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.743679 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.743681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:57.744435 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:57.744437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.744859 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.744860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:57.745016 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.745541 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:57.745549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:57.746852 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:57.748315 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:57.748323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:57.748325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:57.748548 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:57.748821 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:57.748822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:57.748822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:57.748926 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:57.749196 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:57.749203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:57.749334 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 21:09:57.749438 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 21:09:57.749596 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260810 21:09:57.750315 7052 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38643
I20260810 21:09:57.750391 7052 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:57.750571 7052 heartbeater.cc:511] Master 127.27.117.126:38643 requested a full tablet report, sending...
I20260810 21:09:57.750969 6465 ts_manager.cc:194] Registered new tserver with Master: ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66:36243)
I20260810 21:09:57.751351 6465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:47231
WARNING: no policy specified for oryx/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab.
Entry for principal oryx/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab.
I20260810 21:09:57.772447 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38643
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:57.853232 7058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:57.853412 7058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:57.853434 7058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:57.854535 7058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:57.854574 7058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:57.854600 7058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:57.856036 7058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:57.856269 7058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:57.856473 7058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:57.861436 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:57.861441 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.861624 7058 server_base.cc:1034] running on GCE node
W20260810 21:09:57.861442 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:57.861981 7058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:57.862269 7058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:57.863413 7058 hybrid_clock.cc:648] HybridClock initialized: now 1786396197863400 us; error 28 us; skew 500 ppm
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:57.866071 7058 init.cc:377] Logged in from keytab as oryx/127.27.117.67@KRBTEST.COM (short username oryx)
I20260810 21:09:57.866413 7058 webserver.cc:460] Webserver started at http://127.27.117.67:32789/ using document root <none> and password file <none>
I20260810 21:09:57.866609 7058 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:57.866667 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:57.866801 7058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:57.867764 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "f179a38fc7284926ae914ef8c268c16d"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "04f8abfce72f922350021058fa3c1d06"
server_key_iv: "35c47d02b6cb6837e9cf197bfbf32b12"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.868121 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "f179a38fc7284926ae914ef8c268c16d"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "04f8abfce72f922350021058fa3c1d06"
server_key_iv: "35c47d02b6cb6837e9cf197bfbf32b12"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.869714 7058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:57.870447 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.870656 7058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:57.870749 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "f179a38fc7284926ae914ef8c268c16d"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "04f8abfce72f922350021058fa3c1d06"
server_key_iv: "35c47d02b6cb6837e9cf197bfbf32b12"
server_key_version: "encryptionkey@0"
I20260810 21:09:57.870822 7058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:57.878340 7058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:57.880020 7058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:57.880168 7058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:57.880388 7058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:57.880684 7058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:57.880718 7058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.880738 7058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:57.880803 7058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:57.886552 7058 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:45975
I20260810 21:09:57.886622 7286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:45975 every 8 connection(s)
I20260810 21:09:57.886940 7058 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:09:57.887131 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 7058
I20260810 21:09:57.887238 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:09:57.887436 28117 external_mini_cluster.cc:1428] Setting key 2ed281d6cd05b8097a283a72d016372c
Aug 10 21:09:57 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:09:57.896199 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.887703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:52923 (local address 127.27.117.126:38643)
0810 21:09:57.887764 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:57.887766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:57.888074 (+ 308us) server_negotiation.cc:407] Connection header received
0810 21:09:57.888662 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:57.888663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:57.888671 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:57.888718 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:57.889685 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.890174 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.891126 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.891304 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.891354 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.893584 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:57.893586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:57.893586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:57.893593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:57.894782 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.895186 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.895187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.895187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.895248 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:57.895509 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:57.895509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:57.895510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:57.895657 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:57.895730 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:57.896015 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 21:09:57.896111 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:09:57.896368 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:57.887677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:52923)
0810 21:09:57.887855 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:57.887921 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:57.888589 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:57.888773 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:57.888824 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:57.889397 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:57.889601 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:57.889606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.890263 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.890264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:57.891046 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:57.891047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:57.891375 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:57.891376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:57.891477 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:57.892027 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:57.892040 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:57.893469 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:57.894869 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:57.894876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:57.894879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:57.895108 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:57.895317 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:57.895318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:57.895319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:57.895432 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:57.895732 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:57.895739 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:57.895875 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 21:09:57.895977 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 21:09:57.896137 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260810 21:09:57.896886 7287 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38643
I20260810 21:09:57.896991 7287 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:57.897197 7287 heartbeater.cc:511] Master 127.27.117.126:38643 requested a full tablet report, sending...
I20260810 21:09:57.897590 6465 ts_manager.cc:194] Registered new tserver with Master: f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975)
I20260810 21:09:57.897977 6465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:52923
I20260810 21:09:57.899178 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.117.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.125.keytab.
Entry for principal oryx/127.27.117.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.125.keytab.
I20260810 21:09:57.924450 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:39543
--webserver_interface=127.27.117.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.125.keytab
--principal=oryx/127.27.117.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38643,127.27.117.125:39543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:58.006142 7292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:58.006340 7292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:58.006361 7292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:58.007591 7292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:58.007632 7292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:58.007647 7292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:58.007676 7292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:58.007701 7292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:58.009271 7292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38643,127.27.117.125:39543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.125.keytab
--principal=oryx/127.27.117.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.117.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
W20260810 21:09:58.009394 7292 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.117.126:38643,127.27.117.125:39543'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 21:09:58.009547 7292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:58.009783 7292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.014547 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.014603 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.014528 7301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.014736 7292 server_base.cc:1034] running on GCE node
I20260810 21:09:58.014972 7292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.015268 7292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.016428 7292 hybrid_clock.cc:648] HybridClock initialized: now 1786396198016408 us; error 33 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.018873 7292 init.cc:377] Logged in from keytab as oryx/127.27.117.125@KRBTEST.COM (short username oryx)
I20260810 21:09:58.019356 7292 webserver.cc:460] Webserver started at http://127.27.117.125:35097/ using document root <none> and password file <none>
I20260810 21:09:58.019598 7292 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.019644 7292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.019807 7292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:58.020881 7292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/instance:
uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "c573cef0a3ec38297496eaf3e179d0ff"
server_key_iv: "59b74be5dc70d8d9531f8a5d27433b8d"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.021308 7292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal/instance:
uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "c573cef0a3ec38297496eaf3e179d0ff"
server_key_iv: "59b74be5dc70d8d9531f8a5d27433b8d"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.022652 7292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:09:58.023360 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.023554 7292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:58.023651 7292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "c573cef0a3ec38297496eaf3e179d0ff"
server_key_iv: "59b74be5dc70d8d9531f8a5d27433b8d"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.023722 7292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.038884 7292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.040524 7292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:58.040637 7292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.044651 7292 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.125:39543
I20260810 21:09:58.044682 7411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.125:39543 every 8 connection(s)
I20260810 21:09:58.045046 7292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
I20260810 21:09:58.046001 7412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:58.047487 7412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:09:58.049409 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 7292
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.125@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.057224 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.048173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.27.117.126:38643)
0810 21:09:58.048260 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.048263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.048453 (+ 190us) server_negotiation.cc:407] Connection header received
0810 21:09:58.049041 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.049043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.049056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.049103 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.049764 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.050549 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.051862 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.052075 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.052147 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.054199 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.054201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.054201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.054210 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.055716 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.056087 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.056088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.056088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.056162 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.056465 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.056466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.056466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.056626 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.056709 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.057047 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.057131 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:09:58.057405 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.048151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:42018)
0810 21:09:58.048327 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.048371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.048926 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.049169 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.049173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.049431 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.049672 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.049677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.050652 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.050653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.051778 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.051781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.052154 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.052155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.052252 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.052747 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.052755 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.054108 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.055800 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.055808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.055810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.056032 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.056248 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.056250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.056250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.056373 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.056700 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.056707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.056904 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.057016 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.057183 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260810 21:09:58.058176 7412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:09:58.059060 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.049975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.27.117.125:39543)
0810 21:09:58.050105 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.050109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.050166 (+ 57us) server_negotiation.cc:407] Connection header received
0810 21:09:58.050233 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.050234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.050250 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.050313 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.050989 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.051992 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.052744 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.053093 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.053174 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.055836 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.055842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.055843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.055848 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.057516 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.057992 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.057995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.057995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.058059 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.058292 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.058293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.058294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.058624 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.058768 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.058848 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.058954 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.125@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.064810 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.058409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:54970)
0810 21:09:58.058460 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.058470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.058505 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.058757 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.058758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.058927 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.058966 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.058967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.059897 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.059898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.060403 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.060404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.060826 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.060827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.060890 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.061215 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.061220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.062249 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.064042 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.064043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.064043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.064197 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.064356 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.064357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.064357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.064404 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.064573 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.064573 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.064605 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.064654 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.064721 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 21:09:58.064968 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.058500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.27.117.125:39543)
0810 21:09:58.058653 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.058655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.058660 (+ 5us) server_negotiation.cc:407] Connection header received
0810 21:09:58.058674 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.058674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.058681 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.058755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.059123 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.059799 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.060494 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.060707 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.060817 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.062346 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.062347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.062348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.062354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.063973 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.064259 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.064260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.064260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.064303 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.064441 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.064441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.064442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.064523 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.064569 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.064668 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.064763 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260810 21:09:58.066226 7412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Bootstrap starting.
I20260810 21:09:58.066799 7412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:58.067013 7412 log.cc:824] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:58.067716 7412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: No bootstrap required, opened a new log
I20260810 21:09:58.069205 7412 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } }
I20260810 21:09:58.069321 7412 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:58.069343 7412 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0752cc294db4e56b0e12f0dfce8e7cb, State: Initialized, Role: FOLLOWER
I20260810 21:09:58.069428 7412 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } }
I20260810 21:09:58.069667 7425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } } }
I20260810 21:09:58.069731 7412 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:58.069752 7425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: FOLLOWER
W20260810 21:09:58.071486 7448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:09:58.071560 7448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:09:58.073709 7292 master_runner.cc:428] Detected that this master f0752cc294db4e56b0e12f0dfce8e7cb is joining an existing cluster
I20260810 21:09:58.073750 7292 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.117.125:39543
I20260810 21:09:58.081743 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.075056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.27.117.125:39543)
0810 21:09:58.075086 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.075087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.075254 (+ 167us) server_negotiation.cc:407] Connection header received
0810 21:09:58.075297 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.075298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.075307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.075370 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.075685 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.076152 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.077190 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.077499 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.077528 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.078632 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.078633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.078633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.078640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.080450 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.080952 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.080954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.080954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.081013 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.081201 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.081203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.081203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.081331 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.081401 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.081563 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.081649 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 21:09:58.081837 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.074782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.27.117.126:38643)
0810 21:09:58.074827 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.074829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.074972 (+ 143us) server_negotiation.cc:407] Connection header received
0810 21:09:58.075073 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.075074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.075085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.075117 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.075644 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.076409 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.077426 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.077727 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.077798 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.078872 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.078873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.078874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.078882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.080554 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.081015 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.081016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.081016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.081094 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.081265 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.081266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.081267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.081408 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.081472 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.081665 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.081738 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:09:58.082051 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.075015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:54982)
0810 21:09:58.075182 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.075192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.075244 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.075370 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.075370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.075576 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.075630 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.075632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.076239 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.076240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.077107 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.077109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.077594 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.077595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.077630 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.078094 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.078100 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.078530 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.080547 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.080548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.080549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.080878 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.081065 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.081065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.081066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.081120 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.081402 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.081403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.081471 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.081541 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.081971 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":323,"thread_start_us":71,"threads_started":1}
I20260810 21:09:58.082509 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.074743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:42032)
0810 21:09:58.074906 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.074916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.074985 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.075206 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.075207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.075447 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.075546 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.075548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.076493 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.076494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.077341 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.077343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.077854 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.077855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.077930 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.078283 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.078287 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.078782 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.080608 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.080609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.080610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.080892 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.081149 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.081149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.081150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.081214 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.081477 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.081478 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.081515 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.081632 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.081759 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.126@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.091872 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.084707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:54992)
0810 21:09:58.084855 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.084896 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.084974 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.085118 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.085119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.085355 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.085400 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.085402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.086268 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.086269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.086822 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.086824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.087281 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.087282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.087338 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.087757 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.087761 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.089249 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.090896 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.090902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.090904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.091095 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.091300 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.091300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.091300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.091352 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.091600 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.091606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.091645 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.091708 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.091793 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":102,"threads_started":1}
I20260810 21:09:58.091920 7423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.084766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.27.117.125:39543)
0810 21:09:58.084820 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.084822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.084963 (+ 141us) server_negotiation.cc:407] Connection header received
0810 21:09:58.085029 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.085030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.085048 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.085115 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.085469 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.086183 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.086898 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.087200 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.087271 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.089325 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.089326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.089326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.089333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.090808 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.091172 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.091173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.091173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.091232 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.091412 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.091413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.091413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.091535 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.091604 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.091754 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.091834 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:09:58.092489 6465 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.117.125:39543: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
member_type: NON_VOTER
last_known_addr {
host: "127.27.117.125"
port: 39543
}
attrs {
promote: true
}
}
dest_uuid: "a7587b80d5004d33880aa2b91613e611"
cas_config_opid_index: -1
I20260810 21:09:58.092903 6465 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } }
I20260810 21:09:58.093393 7465 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f0752cc294db4e56b0e12f0dfce8e7cb (127.27.117.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } }
I20260810 21:09:58.093683 7362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:58.094338 7465 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.117.125:39543
I20260810 21:09:58.094424 7465 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } }
I20260810 21:09:58.094507 7465 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:58.094625 7362 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Refusing update from remote peer a7587b80d5004d33880aa2b91613e611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 21:09:58.094692 7292 master.cc:537] Master@127.27.117.125:39543 shutting down...
I20260810 21:09:58.095139 7468 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 21:09:58.095791 6434 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 21:09:58.096342 6434 consensus_peers.cc:487] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 -> Peer f0752cc294db4e56b0e12f0dfce8e7cb (127.27.117.125:39543): Couldn't send request to peer f0752cc294db4e56b0e12f0dfce8e7cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 21:09:58.097617 7292 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:58.097707 7292 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:58.097756 7292 tablet_replica.cc:330] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: stopping tablet replica
I20260810 21:09:58.110754 7292 master.cc:552] Master@127.27.117.125:39543 shutdown complete.
I20260810 21:09:58.110816 7292 master_runner.cc:305] Clearing existing system tablet
I20260810 21:09:58.110947 7292 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.111011 7292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.111608 7292 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:58.112188 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.112345 7292 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.112404 7292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.112429 7292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "c573cef0a3ec38297496eaf3e179d0ff"
server_key_iv: "59b74be5dc70d8d9531f8a5d27433b8d"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.113214 7292 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:58.114497 7292 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 21:09:58.114555 7292 log.cc:1196] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 21:09:58.114768 7292 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Deleting consensus metadata
I20260810 21:09:58.114897 7292 master_runner.cc:315] Copying system tablet from 127.27.117.126:38643
I20260810 21:09:58.115514 7292 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: tablet copy: Beginning tablet copy session from remote peer at address 127.27.117.126:38643
I20260810 21:09:58.121308 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.115886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.27.117.126:38643)
0810 21:09:58.115934 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.115936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.116001 (+ 65us) server_negotiation.cc:407] Connection header received
0810 21:09:58.116072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.116073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.116083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.116137 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.116426 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.116962 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.117658 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.117855 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.117948 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.118849 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.118850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.118851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.118858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.120126 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.120493 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.120494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.120495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.120558 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.120765 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.120766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.120767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.120904 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.120996 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.121155 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.121223 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:09:58.121327 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.115837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:42044)
0810 21:09:58.115954 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.115963 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.116016 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.116141 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.116141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.116302 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.116341 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.116343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.117043 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.117044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.117576 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.117578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.117947 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.117948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.118019 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.118480 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.118484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.118759 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.120209 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.120210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.120211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.120401 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.120636 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.120637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.120638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.120695 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.120990 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.120991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.121036 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.121117 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.121219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260810 21:09:58.121631 6503 tablet_copy_service.cc:140] P a7587b80d5004d33880aa2b91613e611: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f0752cc294db4e56b0e12f0dfce8e7cb ({username='oryx', principal='oryx/127.27.117.125@KRBTEST.COM'} at 127.0.0.1:42044)
I20260810 21:09:58.121753 6503 tablet_copy_service.cc:161] P a7587b80d5004d33880aa2b91613e611: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f0752cc294db4e56b0e12f0dfce8e7cb at {username='oryx', principal='oryx/127.27.117.125@KRBTEST.COM'} at 127.0.0.1:42044: session id = f0752cc294db4e56b0e12f0dfce8e7cb-00000000000000000000000000000000
I20260810 21:09:58.122402 6503 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 21:09:58.122968 7292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 21:09:58.123872 7292 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: tablet copy: Starting download of 0 data blocks...
I20260810 21:09:58.124028 7292 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: tablet copy: Starting download of 1 WAL segments...
I20260810 21:09:58.124939 7292 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 21:09:58.125775 6503 tablet_copy_service.cc:342] P a7587b80d5004d33880aa2b91613e611: Request end of tablet copy session f0752cc294db4e56b0e12f0dfce8e7cb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.117.125@KRBTEST.COM'} at 127.0.0.1:42044
I20260810 21:09:58.125833 6503 tablet_copy_service.cc:433] P a7587b80d5004d33880aa2b91613e611: ending tablet copy session f0752cc294db4e56b0e12f0dfce8e7cb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f0752cc294db4e56b0e12f0dfce8e7cb
I20260810 21:09:58.128202 7292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.129310 7483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.129321 7484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.129550 7486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.129621 7292 server_base.cc:1034] running on GCE node
I20260810 21:09:58.129725 7292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.129846 7292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.130955 7292 hybrid_clock.cc:648] HybridClock initialized: now 1786396198130943 us; error 27 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.132184 7292 init.cc:377] Logged in from keytab as oryx/127.27.117.125@KRBTEST.COM (short username oryx)
I20260810 21:09:58.132472 7292 webserver.cc:460] Webserver started at http://127.27.117.125:38809/ using document root <none> and password file <none>
I20260810 21:09:58.132589 7292 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.132615 7292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.133232 7292 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:58.133692 7497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.133813 7292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.133867 7292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "c573cef0a3ec38297496eaf3e179d0ff"
server_key_iv: "59b74be5dc70d8d9531f8a5d27433b8d"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.133908 7292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.139837 7292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.141212 7292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.144260 7292 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.125:39543
I20260810 21:09:58.144616 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.125:39543 every 8 connection(s)
I20260810 21:09:58.144686 7292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
I20260810 21:09:58.145601 7585 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:09:58.146113 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Bootstrap starting.
I20260810 21:09:58.149127 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:09:58.149374 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: Bootstrap complete.
I20260810 21:09:58.149580 7585 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } }
I20260810 21:09:58.149672 7585 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f0752cc294db4e56b0e12f0dfce8e7cb, State: Initialized, Role: LEARNER
I20260810 21:09:58.149724 7585 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } }
I20260810 21:09:58.149915 7585 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:58.149915 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } }
I20260810 21:09:58.150098 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: LEARNER
I20260810 21:09:58.151439 7610 catalog_manager.cc:1259] Loaded cluster ID: e3f5693b3d654f8498d63913b10dcb90
I20260810 21:09:58.151727 7610 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: loading cluster ID for follower catalog manager: success
I20260810 21:09:58.152489 7610 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: acquiring CA information for follower catalog manager: success
I20260810 21:09:58.152697 7610 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:09:58.167603 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.160716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.27.117.125:39543)
0810 21:09:58.160841 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.160844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.160854 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:58.160878 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.160879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.160890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.160977 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.161585 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.162587 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.163250 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.163548 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.163620 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.164768 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.164769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.164769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.164777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.166496 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.166827 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.166828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.166829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.166896 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.167087 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.167088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.167089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.167221 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.167328 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.167411 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.167492 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
WARNING: no policy specified for oryx/127.27.117.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.124.keytab.
Entry for principal oryx/127.27.117.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.124.keytab.
I20260810 21:09:58.193434 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:41067
--webserver_interface=127.27.117.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.124.keytab
--principal=oryx/127.27.117.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:58.279662 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:58.279881 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:58.279919 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:58.281085 7616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:09:58.281150 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:58.281174 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:58.281198 7616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:09:58.281226 7616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:09:58.282696 7616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.124.keytab
--principal=oryx/127.27.117.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:41067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.117.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:58.282984 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:58.283209 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.288102 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.288162 7616 server_base.cc:1034] running on GCE node
W20260810 21:09:58.288086 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.288075 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.288462 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.288759 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.289969 7616 hybrid_clock.cc:648] HybridClock initialized: now 1786396198289970 us; error 38 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.292416 7616 init.cc:377] Logged in from keytab as oryx/127.27.117.124@KRBTEST.COM (short username oryx)
I20260810 21:09:58.292888 7616 webserver.cc:460] Webserver started at http://127.27.117.124:45669/ using document root <none> and password file <none>
I20260810 21:09:58.293115 7616 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.293182 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.293336 7616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:09:58.294299 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/instance:
uuid: "12afdaf0442046bd9c39b48a2ecf9450"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "2baafc3de5174c2d510c938ce48fa591"
server_key_iv: "02d0d5fd16eee7b790c1019bc0d2024c"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.294663 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal/instance:
uuid: "12afdaf0442046bd9c39b48a2ecf9450"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "2baafc3de5174c2d510c938ce48fa591"
server_key_iv: "02d0d5fd16eee7b790c1019bc0d2024c"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.295948 7616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:09:58.296612 7640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.296821 7616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:58.296913 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
uuid: "12afdaf0442046bd9c39b48a2ecf9450"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "2baafc3de5174c2d510c938ce48fa591"
server_key_iv: "02d0d5fd16eee7b790c1019bc0d2024c"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.297001 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.316879 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.319036 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:58.319191 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.323097 7616 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.124:41067
I20260810 21:09:58.323144 7742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.124:41067 every 8 connection(s)
I20260810 21:09:58.323505 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
I20260810 21:09:58.324443 7745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:58.325996 7745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:09:58.329046 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 7616
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.124@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.117.124@KRBTEST.COM
I20260810 21:09:58.335516 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.326855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.27.117.126:38643)
0810 21:09:58.326920 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.326922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.326934 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:09:58.327438 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.327440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.327452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.327536 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.328121 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.328631 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.329733 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.330045 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.330142 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.332227 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.332228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.332229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.332236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.334081 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.334555 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.334556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.334556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.334610 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.334879 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.334880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.334880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.335029 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.335104 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.335347 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.335429 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:09:58.335675 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.326573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:42056)
0810 21:09:58.326807 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.326854 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.327307 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.327611 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.327614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.327838 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.328028 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.328032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.328725 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.328726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.329638 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.329641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.330173 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.330175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.330297 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.330764 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.330772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.332136 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.334176 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.334184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.334186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.334482 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.334680 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.334680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.334681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.334822 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.335100 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.335105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.335208 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.335319 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.335459 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260810 21:09:58.336189 7745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:09:58.338474 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.329622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.27.117.124:41067)
0810 21:09:58.329830 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.329834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.329854 (+ 20us) server_negotiation.cc:407] Connection header received
0810 21:09:58.329874 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.329875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.329893 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.329963 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.330511 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.331235 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.332033 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.332335 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.332412 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.335256 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.335263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.335264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.335270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.337445 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.337749 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.337752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.337753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.337817 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.338033 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.338033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.338034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.338150 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.338249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.338356 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.338412 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.124@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.342880 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.336432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:55030)
0810 21:09:58.336493 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.336504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.336543 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.336857 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.336858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.336974 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.336999 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.337000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.337885 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.337887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.338668 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.338671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.339020 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.339022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.339083 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.339375 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.339378 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.340471 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.341901 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.341902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.341902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.342071 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.342274 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.342275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.342276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.342337 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.342581 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.342582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.342623 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.342708 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.342795 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 21:09:58.342923 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.336629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.27.117.125:39543)
0810 21:09:58.336703 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.336705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.336716 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:58.336743 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.336744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.336759 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.336842 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.337085 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.337781 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.338748 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.338943 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.339023 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.340579 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.340580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.340581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.340588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.341798 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.342147 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.342148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.342148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.342207 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.342409 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.342410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.342410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.342514 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.342588 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.342748 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.342830 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:09:58.343321 7745 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.124@KRBTEST.COM for oryx/127.27.117.124@KRBTEST.COM
I20260810 21:09:58.349066 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.343536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:37054)
0810 21:09:58.343582 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.343592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.343632 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.343750 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.343750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.343875 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.343898 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.343899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.344480 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.344481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.345057 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.345064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.345427 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.345427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.345474 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.345780 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.345784 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.346789 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.348054 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.348054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.348055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.348213 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.348427 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.348427 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.348428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.348486 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.348742 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.348743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.348778 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.348876 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.348963 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 21:09:58.349138 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.343584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.27.117.124:41067)
0810 21:09:58.343616 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.343618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.343626 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:58.343686 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.343687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.343695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.343747 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.343970 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.344405 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.345148 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.345348 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.345426 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.346869 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.346870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.346871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.346877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.347956 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.348298 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.348299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.348299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.348353 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.348574 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.348575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.348575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.348675 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.348749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.348900 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.349013 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":18}
I20260810 21:09:58.350487 7745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Bootstrap starting.
I20260810 21:09:58.351287 7745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:58.351573 7745 log.cc:824] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:58.352180 7745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: No bootstrap required, opened a new log
I20260810 21:09:58.353605 7745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } }
I20260810 21:09:58.353716 7745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:58.353739 7745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12afdaf0442046bd9c39b48a2ecf9450, State: Initialized, Role: FOLLOWER
I20260810 21:09:58.353850 7745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } }
I20260810 21:09:58.354089 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } } }
I20260810 21:09:58.354140 7745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:58.354188 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 21:09:58.356163 7770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:09:58.356226 7770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:09:58.358420 7616 master_runner.cc:428] Detected that this master 12afdaf0442046bd9c39b48a2ecf9450 is joining an existing cluster
I20260810 21:09:58.358474 7616 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.117.124:41067
I20260810 21:09:58.366451 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.359900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.27.117.126:38643)
0810 21:09:58.359942 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.359943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.359951 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:58.359986 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.359987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.359996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.360069 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.360533 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.361312 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362496 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.362790 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362827 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.363858 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.363859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.363860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.363866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.365277 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.365651 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.365652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.365652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.365708 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.365928 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.365929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.365929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.366044 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.366117 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.366287 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.366359 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:09:58.366477 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.359681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:42068)
0810 21:09:58.359866 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.359877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.359928 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.360075 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.360075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.360296 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.360385 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.360387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.361397 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.361398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.362391 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.362393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.362916 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.362972 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.363416 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.363421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.363737 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.365379 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.365380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.365380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.365558 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.365794 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.365794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.365795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.365853 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.366120 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.366121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.366164 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.366261 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.366361 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":34,"thread_start_us":110,"threads_started":1}
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396197, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.126@KRBTEST.COM for oryx/127.27.117.124@KRBTEST.COM
I20260810 21:09:58.380872 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.360217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.27.117.124:41067)
0810 21:09:58.360256 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.360258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.360462 (+ 204us) server_negotiation.cc:407] Connection header received
0810 21:09:58.360542 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.360543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.360551 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.360575 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.361026 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.361652 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362587 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.362846 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362881 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.378038 (+ 15157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.378039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.378040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.378046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.379741 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380092 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.380093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.380093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.380139 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380308 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.380309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.380309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.380415 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.380481 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.380663 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.380734 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:09:58.380878 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.368248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:37084)
0810 21:09:58.368310 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.368322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.368367 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.368615 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.368616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.368735 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.368766 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.368767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.369720 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.369721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.370203 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.370204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.370682 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.370682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.370739 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.371128 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.371132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.372258 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.379639 (+ 7381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.379640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.379641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.379850 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.380089 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.380091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.380151 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.380483 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.380485 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.380533 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.380586 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.380701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 21:09:58.381310 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.368299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.27.117.124:41067)
0810 21:09:58.368461 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.368463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.368471 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:58.368504 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.368505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.368515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.368606 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.368880 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.369646 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.370277 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.370606 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.370675 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.372350 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.372351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.372352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.372358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.379537 (+ 7179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.379939 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.379940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.379940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.380009 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380227 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.380227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.380228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.380391 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.381076 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.381116 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.381199 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260810 21:09:58.381451 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.359679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:55044)
0810 21:09:58.359850 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.359864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.359969 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.360155 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.360156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.360385 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.360446 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.360448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.361454 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.361455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.362391 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.362394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.362924 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.362991 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.377245 (+ 14254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.377250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.377691 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.380227 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.380228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.380505 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.380720 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.380721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.380812 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.381154 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.381155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.381193 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.381254 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.381345 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":37,"spinlock_wait_cycles":8960,"thread_start_us":111,"threads_started":1}
I20260810 21:09:58.381755 6465 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.117.124:41067: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450"
member_type: NON_VOTER
last_known_addr {
host: "127.27.117.124"
port: 41067
}
attrs {
promote: true
}
}
dest_uuid: "a7587b80d5004d33880aa2b91613e611"
cas_config_opid_index: 5
I20260810 21:09:58.381821 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.359727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.27.117.125:39543)
0810 21:09:58.359771 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.359773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.359930 (+ 157us) server_negotiation.cc:407] Connection header received
0810 21:09:58.360054 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.360056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.360067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.360096 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.360550 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.361324 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362498 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.362805 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362847 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.377797 (+ 14950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.377799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.377799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.377808 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.380141 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380587 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.380588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.380589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.380648 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380882 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.380883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.380884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.381030 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.381092 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.381643 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.381735 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:09:58.382018 6465 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } }
I20260810 21:09:58.382362 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.360271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:37070)
0810 21:09:58.360409 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.360417 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.360482 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.360618 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.360619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.360849 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.360903 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.360905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.361731 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.361731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.362517 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.362519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.362941 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.362941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.363002 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.377449 (+ 14447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.377454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.377932 (+ 478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.379827 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.379828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.379828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.380020 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.380202 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.380202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.380203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.380249 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.380519 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.380520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.380564 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.381692 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.381793 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260810 21:09:58.382388 7468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.383332 7468 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.117.124:41067
I20260810 21:09:58.383427 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.383508 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:58.383599 7616 master.cc:537] Master@127.27.117.124:41067 shutting down...
W20260810 21:09:58.384428 6438 consensus_peers.cc:487] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 -> Peer 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124:41067): Couldn't send request to peer 12afdaf0442046bd9c39b48a2ecf9450. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 21:09:58.386876 7616 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:58.386972 7616 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:58.387025 7616 tablet_replica.cc:330] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: stopping tablet replica
I20260810 21:09:58.387404 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.383827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.27.117.125:39543)
0810 21:09:58.384239 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.384241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.384252 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:58.384315 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.384316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.384340 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.384399 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:58.384573 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.385295 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.386469 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.387097 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.387167 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.387247 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.387311 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.387343 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260810 21:09:58.387406 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.383722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:55060)
0810 21:09:58.384218 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.384230 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.384265 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.384471 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.384472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.384474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:58.384514 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.384515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.385390 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.385393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.386359 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.386362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.387165 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.387166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.387221 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.387242 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.387304 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.387341 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481}
I20260810 21:09:58.387665 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 LEARNER]: Refusing update from remote peer a7587b80d5004d33880aa2b91613e611: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 21:09:58.387843 7465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 21:09:58.388770 7538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.389928 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.390082 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: LEARNER
I20260810 21:09:58.390169 7468 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: attempting to promote NON_VOTER f0752cc294db4e56b0e12f0dfce8e7cb to VOTER
I20260810 21:09:58.390399 7468 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } }
I20260810 21:09:58.390175 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7587b80d5004d33880aa2b91613e611. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: NON_VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: true } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.390841 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: LEARNER
I20260810 21:09:58.390893 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.390791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:37102)
0810 21:09:58.390827 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.390868 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 21:09:58.391155 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 LEARNER]: Refusing update from remote peer a7587b80d5004d33880aa2b91613e611: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 21:09:58.391249 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.391169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:37112)
0810 21:09:58.391199 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.391226 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260810 21:09:58.391343 7796 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260810 21:09:58.391412 6438 consensus_peers.cc:487] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 -> Peer 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124:41067): Couldn't send request to peer 12afdaf0442046bd9c39b48a2ecf9450. Status: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 21:09:58.391980 7465 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0752cc294db4e56b0e12f0dfce8e7cb (127.27.117.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.392696 7465 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.392767 7465 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:58.392850 7465 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.392902 7465 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:58.392947 7538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f0752cc294db4e56b0e12f0dfce8e7cb (127.27.117.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.393971 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.394076 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:09:58.401839 7616 master.cc:552] Master@127.27.117.124:41067 shutdown complete.
I20260810 21:09:58.401893 7616 master_runner.cc:305] Clearing existing system tablet
I20260810 21:09:58.402025 7616 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.402077 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.402791 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:58.403297 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.403437 7616 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.403501 7616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.403527 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
uuid: "12afdaf0442046bd9c39b48a2ecf9450"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "2baafc3de5174c2d510c938ce48fa591"
server_key_iv: "02d0d5fd16eee7b790c1019bc0d2024c"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.404286 7616 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:58.405643 7616 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 21:09:58.405717 7616 log.cc:1196] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 21:09:58.405918 7616 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Deleting consensus metadata
I20260810 21:09:58.406035 7616 master_runner.cc:315] Copying system tablet from 127.27.117.126:38643
I20260810 21:09:58.406744 7616 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: tablet copy: Beginning tablet copy session from remote peer at address 127.27.117.126:38643
I20260810 21:09:58.413136 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.407109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.27.117.126:38643)
0810 21:09:58.407171 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.407173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.407292 (+ 119us) server_negotiation.cc:407] Connection header received
0810 21:09:58.407359 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.407360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.407371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.407443 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.407784 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.408354 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.409194 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.409395 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.409483 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.410431 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.410432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.410433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.410440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.411984 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.412309 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.412310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.412311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.412368 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.412597 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.412598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.412598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.412712 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.412789 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.412975 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.413045 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:09:58.413151 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.407043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:42072)
0810 21:09:58.407217 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.407234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.407308 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.407441 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.407442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.407622 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.407716 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.407719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.408436 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.408437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.409101 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.409103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.409484 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.409485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.409559 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.410056 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.410060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.410340 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.412060 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.412061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.412062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.412236 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.412445 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.412446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.412446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.412530 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.412803 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.412804 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.412850 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.412938 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.413046 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260810 21:09:58.413648 6503 tablet_copy_service.cc:140] P a7587b80d5004d33880aa2b91613e611: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 12afdaf0442046bd9c39b48a2ecf9450 ({username='oryx', principal='oryx/127.27.117.124@KRBTEST.COM'} at 127.0.0.1:42072)
I20260810 21:09:58.413715 6503 tablet_copy_service.cc:161] P a7587b80d5004d33880aa2b91613e611: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 12afdaf0442046bd9c39b48a2ecf9450 at {username='oryx', principal='oryx/127.27.117.124@KRBTEST.COM'} at 127.0.0.1:42072: session id = 12afdaf0442046bd9c39b48a2ecf9450-00000000000000000000000000000000
I20260810 21:09:58.414134 6503 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 21:09:58.414604 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 21:09:58.415545 7616 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: tablet copy: Starting download of 0 data blocks...
I20260810 21:09:58.415748 7616 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: tablet copy: Starting download of 1 WAL segments...
I20260810 21:09:58.416591 7616 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 21:09:58.417443 6503 tablet_copy_service.cc:342] P a7587b80d5004d33880aa2b91613e611: Request end of tablet copy session 12afdaf0442046bd9c39b48a2ecf9450-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.117.124@KRBTEST.COM'} at 127.0.0.1:42072
I20260810 21:09:58.417511 6503 tablet_copy_service.cc:433] P a7587b80d5004d33880aa2b91613e611: ending tablet copy session 12afdaf0442046bd9c39b48a2ecf9450-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 12afdaf0442046bd9c39b48a2ecf9450
I20260810 21:09:58.420185 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.421319 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.421365 7808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.421501 7616 server_base.cc:1034] running on GCE node
W20260810 21:09:58.425231 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.425410 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.425561 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.426683 7616 hybrid_clock.cc:648] HybridClock initialized: now 1786396198426676 us; error 23 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.428050 7616 init.cc:377] Logged in from keytab as oryx/127.27.117.124@KRBTEST.COM (short username oryx)
I20260810 21:09:58.428344 7616 webserver.cc:460] Webserver started at http://127.27.117.124:34359/ using document root <none> and password file <none>
I20260810 21:09:58.428480 7616 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.428532 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.429204 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:58.429741 7825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.429898 7616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:09:58.429965 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
uuid: "12afdaf0442046bd9c39b48a2ecf9450"
format_stamp: "Formatted at 2026-08-10 21:09:58 on dist-test-slave-2kcd"
server_key: "2baafc3de5174c2d510c938ce48fa591"
server_key_iv: "02d0d5fd16eee7b790c1019bc0d2024c"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.430022 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.454790 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.456223 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.460438 7616 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.124:41067
I20260810 21:09:58.460455 7922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.124:41067 every 8 connection(s)
I20260810 21:09:58.460774 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
I20260810 21:09:58.461467 7923 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:09:58.461921 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Bootstrap starting.
I20260810 21:09:58.464111 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:09:58.464349 7923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Bootstrap complete.
I20260810 21:09:58.464533 7923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } }
I20260810 21:09:58.464620 7923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12afdaf0442046bd9c39b48a2ecf9450, State: Initialized, Role: LEARNER
I20260810 21:09:58.464671 7923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } }
I20260810 21:09:58.464870 7923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:09:58.464833 7929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.464928 7929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: This master's current role is: LEARNER
I20260810 21:09:58.466291 7954 catalog_manager.cc:1259] Loaded cluster ID: e3f5693b3d654f8498d63913b10dcb90
I20260810 21:09:58.466362 7954 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: loading cluster ID for follower catalog manager: success
I20260810 21:09:58.467392 7954 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: acquiring CA information for follower catalog manager: success
I20260810 21:09:58.467660 7954 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:09:58.497287 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.490501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.27.117.124:41067)
0810 21:09:58.490618 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.490620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.490628 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:58.490647 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.490648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.490655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.490721 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.491458 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.492120 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.493167 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.493364 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.493414 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.494859 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.494860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.494860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.494866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.496217 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.496609 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.496610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.496610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.496659 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.496870 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.496871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.496871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.496977 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.497051 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.497169 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.497225 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260810 21:09:58.497876 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 6587
I20260810 21:09:58.512264 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:36283
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:38643
--webserver_port=41839
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:58.592693 7959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:58.592908 7959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:58.592932 7959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:58.593940 7959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:58.593971 7959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:58.593997 7959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:09:58.595322 7959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:36283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=41839
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:58.595500 7959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:58.595666 7959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.600750 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.600719 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.600729 7969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.600829 7959 server_base.cc:1034] running on GCE node
I20260810 21:09:58.601100 7959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.601318 7959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.602484 7959 hybrid_clock.cc:648] HybridClock initialized: now 1786396198602467 us; error 28 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.605047 7959 init.cc:377] Logged in from keytab as oryx/127.27.117.65@KRBTEST.COM (short username oryx)
I20260810 21:09:58.605450 7959 webserver.cc:460] Webserver started at http://127.27.117.65:41839/ using document root <none> and password file <none>
I20260810 21:09:58.605665 7959 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.605857 7959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.607446 7959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:09:58.608086 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.608280 7959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:09:58.608394 7959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "48ecce09cf3248f2bddc714750d40946"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "77d329bac7aab88c561271bd62bffa9f"
server_key_iv: "55cecd0b7a37aed10490693617a34709"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.608722 7959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.630947 7959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.633387 7959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:58.633562 7959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.633808 7959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:58.634135 7959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:58.634189 7959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.634229 7959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:58.634258 7959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.640401 7959 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:36283
I20260810 21:09:58.640437 8180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:36283 every 8 connection(s)
I20260810 21:09:58.640857 7959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:09:58.647279 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 7959
I20260810 21:09:58.647464 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 6824
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.124@KRBTEST.COM
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.655285 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.641608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:40339 (local address 127.27.117.124:41067)
0810 21:09:58.641668 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.641671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.641929 (+ 258us) server_negotiation.cc:407] Connection header received
0810 21:09:58.642968 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.642970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.642985 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.643040 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.643913 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.644650 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.645792 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.646012 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.646043 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.649011 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.649012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.649012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.649019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.653540 (+ 4521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.654002 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.654004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.654004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.654072 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.654373 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.654374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.654375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.654558 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.654642 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.655028 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.655137 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:09:58.655702 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.641631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.27.117.65:40339)
0810 21:09:58.641805 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.641849 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.642866 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.643086 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.643095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.643577 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.643804 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.643810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.644741 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.644741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.645688 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.645691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.646104 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.646104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.646149 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.646519 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.646523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.648909 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.653630 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.653636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.653638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.653924 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.654146 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.654147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.654147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.654281 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.654645 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.654651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.654820 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.654946 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.655183 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":50,"thread_start_us":88,"threads_started":1}
I20260810 21:09:58.656250 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.641773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:49317 (local address 127.27.117.125:39543)
0810 21:09:58.641882 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.641885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.642145 (+ 260us) server_negotiation.cc:407] Connection header received
0810 21:09:58.642962 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.642964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.642975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.643019 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.643897 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.644376 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.645439 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.645648 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.645724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.649590 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.649592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.649592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.649600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.654959 (+ 5359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.655407 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.655408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.655409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.655478 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.655671 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.655672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.655672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.655824 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.655874 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.656092 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.656166 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 21:09:58.656263 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.641634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.27.117.65:49317)
0810 21:09:58.642084 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.642094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.642880 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.643086 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.643094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.643528 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.643804 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.643808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.644482 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.644483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.645349 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.645351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.645730 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.645731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.645866 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.646362 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.646372 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.649509 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.655054 (+ 5545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.655055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.655055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.655282 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.655548 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.655548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.655549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.655603 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.655925 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.655927 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.655967 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.656053 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.656157 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":2,"spinlock_wait_cycles":7040,"thread_start_us":113,"threads_started":1}
I20260810 21:09:58.656350 8182 heartbeater.cc:347] Connected to a master server at 127.27.117.124:41067
I20260810 21:09:58.656481 8182 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.656733 8182 heartbeater.cc:511] Master 127.27.117.124:41067 requested a full tablet report, sending...
I20260810 21:09:58.657474 7858 ts_manager.cc:194] Registered new tserver with Master: 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:09:58.657672 8185 heartbeater.cc:347] Connected to a master server at 127.27.117.125:39543
I20260810 21:09:58.657734 8185 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.657984 8185 heartbeater.cc:511] Master 127.27.117.125:39543 requested a full tablet report, sending...
I20260810 21:09:58.658433 7525 ts_manager.cc:194] Registered new tserver with Master: 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:09:58.681028 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.642161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:49365 (local address 127.27.117.126:38643)
0810 21:09:58.642228 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.642230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.643152 (+ 922us) server_negotiation.cc:407] Connection header received
0810 21:09:58.643213 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.643215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.643245 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.643317 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.643914 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.644642 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.645815 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.646608 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.646686 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.669041 (+ 22355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.669043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.669044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.669052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.679814 (+ 10762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.680201 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.680203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.680203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.680265 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.680465 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.680465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.680466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.680597 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.680674 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.680860 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.680936 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:09:58.681042 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.641940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:49365)
0810 21:09:58.643076 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.643085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.643146 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.643332 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.643333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.643577 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.643804 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.643810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.644741 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.644741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.645688 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.645691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.646697 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.646698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.646767 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.667396 (+ 20629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.667402 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.668932 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.679922 (+ 10990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.679924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.679924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.680112 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.680338 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.680339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.680339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.680397 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.680664 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.680665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.680712 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.680811 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.680934 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":89,"thread_start_us":88,"threads_started":1}
I20260810 21:09:58.681427 8184 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38643
I20260810 21:09:58.681486 8184 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.681576 8184 heartbeater.cc:511] Master 127.27.117.126:38643 requested a full tablet report, sending...
I20260810 21:09:58.681902 6465 ts_manager.cc:194] Re-registered known tserver with Master: 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:09:58.682403 6465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:49365
I20260810 21:09:58.683532 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:36243
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:38643
--webserver_port=44857
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:09:58.767051 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:58.767279 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:58.767316 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:58.768428 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:58.768489 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:58.768535 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:09:58.770097 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:36243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=44857
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:58.770357 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:58.770598 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.775703 8205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.775776 8196 server_base.cc:1034] running on GCE node
W20260810 21:09:58.775691 8215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.775682 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.776103 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.776369 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.777530 8196 hybrid_clock.cc:648] HybridClock initialized: now 1786396198777521 us; error 38 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.779979 8196 init.cc:377] Logged in from keytab as oryx/127.27.117.66@KRBTEST.COM (short username oryx)
I20260810 21:09:58.780450 8196 webserver.cc:460] Webserver started at http://127.27.117.66:44857/ using document root <none> and password file <none>
I20260810 21:09:58.780628 8196 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.780679 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.781945 8196 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:58.782451 8228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.782629 8196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.782719 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "ff0643f61c584a53a0eda4c92231ba22"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "8f153efcedaad54cec2287ea17710b10"
server_key_iv: "f4cd38ce243e8db5dabe779ce39c5dc3"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.783037 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.800094 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.796294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:37140)
0810 21:09:58.796349 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.796358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.796396 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.796634 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.796636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.796638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:58.796694 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.796697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.797728 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.797729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.798928 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.798930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.799712 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.799714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.799866 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.799872 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.799979 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.800033 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 21:09:58.800103 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.796403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.27.117.124:41067)
0810 21:09:58.796453 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.796456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.796466 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:58.796497 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.796498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.796525 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.796627 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:58.796852 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.797629 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.799022 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.799613 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.799681 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.799744 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.799990 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.800034 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:09:58.800576 7878 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 21:09:58.800774 7929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7587b80d5004d33880aa2b91613e611. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:58.800890 7929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: This master's current role is: LEARNER
I20260810 21:09:58.821272 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.823993 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:58.824172 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.825367 8196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:58.825707 8196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:58.825742 8196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.825765 8196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:58.825778 8196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.832242 8196 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36243
I20260810 21:09:58.832304 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36243 every 8 connection(s)
I20260810 21:09:58.832657 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:09:58.839387 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 8196
I20260810 21:09:58.839560 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 7058
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.854604 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.833742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:35829 (local address 127.27.117.126:38643)
0810 21:09:58.833796 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.833798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.834249 (+ 451us) server_negotiation.cc:407] Connection header received
0810 21:09:58.835310 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.835311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.835325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.835376 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.836110 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.836616 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.837989 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.838279 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838329 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.850729 (+ 12400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.850731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.850732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.850740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.852574 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.853302 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.853303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.853304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.853377 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.853768 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.853769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.853770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.853928 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.853973 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.854398 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.854499 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:09:58.854893 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.833489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:35829)
0810 21:09:58.834150 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.834167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.835202 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.835442 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.835446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.835812 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.836007 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.836012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.836727 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.836728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.837902 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.837905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.838353 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.838421 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.838926 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.838934 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.850607 (+ 11673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.852696 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.852707 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.852710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.853188 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.853474 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.853475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.853475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.853673 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.854021 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.854028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.854206 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.854317 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.854511 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":10,"thread_start_us":379,"threads_started":1}
I20260810 21:09:58.855378 8419 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38643
I20260810 21:09:58.855481 8419 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.855685 8419 heartbeater.cc:511] Master 127.27.117.126:38643 requested a full tablet report, sending...
I20260810 21:09:58.856632 6465 ts_manager.cc:194] Re-registered known tserver with Master: ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66:36243)
I20260810 21:09:58.857167 6465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:35829
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.124@KRBTEST.COM
I20260810 21:09:58.857712 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.834027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:43705 (local address 127.27.117.125:39543)
0810 21:09:58.834073 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.834075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.834438 (+ 363us) server_negotiation.cc:407] Connection header received
0810 21:09:58.835318 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.835319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.835333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.835367 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.836121 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.836909 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838376 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.838564 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838612 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.854895 (+ 16283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.854896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.854897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.854905 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.856439 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.856883 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.856884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.856884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.856943 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.857152 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.857152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.857153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.857282 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.857327 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.857540 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.857624 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:09:58.858031 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.833955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.27.117.66:43705)
0810 21:09:58.834374 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.834382 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.835218 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.835460 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.835460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.835885 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.836007 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.836014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.836984 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.836985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.837906 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.837908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.838643 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.838708 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.839063 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.839069 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.854787 (+ 15718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.856538 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.856540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.856540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.856765 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.857031 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.857032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.857032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.857087 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.857379 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.857380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.857423 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.857825 (+ 402us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.857924 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260810 21:09:58.858552 8428 heartbeater.cc:347] Connected to a master server at 127.27.117.125:39543
I20260810 21:09:58.858595 8428 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.858639 8428 heartbeater.cc:511] Master 127.27.117.125:39543 requested a full tablet report, sending...
I20260810 21:09:58.858945 7525 ts_manager.cc:194] Registered new tserver with Master: ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66:36243)
I20260810 21:09:58.859627 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:45975
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:38643
--webserver_port=32789
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:09:58.861090 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.833465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:41837 (local address 127.27.117.124:41067)
0810 21:09:58.833526 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.833529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.834247 (+ 718us) server_negotiation.cc:407] Connection header received
0810 21:09:58.835318 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.835319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.835332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.835367 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.836119 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.836915 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.837824 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.838093 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838163 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.857764 (+ 19601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.857766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.857766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.857775 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.859888 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.860281 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.860282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.860282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.860332 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.860534 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.860541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.860541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.860640 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.860691 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.860892 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.861002 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:09:58.861155 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.833489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.27.117.66:41837)
0810 21:09:58.834119 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.834167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.835219 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.835446 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.835447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.835894 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.836007 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.836014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.837009 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.837010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.837732 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.837733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.838168 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.838168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.838262 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.855634 (+ 17372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.855640 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.857643 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.859973 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.859974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.859974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.860203 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.860400 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.860401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.860402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.860473 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.860704 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.860705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.860750 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.860853 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.860993 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":3,"thread_start_us":351,"threads_started":1}
I20260810 21:09:58.861531 8416 heartbeater.cc:347] Connected to a master server at 127.27.117.124:41067
I20260810 21:09:58.861588 8416 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.861642 8416 heartbeater.cc:511] Master 127.27.117.124:41067 requested a full tablet report, sending...
I20260810 21:09:58.861938 7858 ts_manager.cc:194] Registered new tserver with Master: ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66:36243)
W20260810 21:09:58.941565 8433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:09:58.941741 8433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:09:58.941762 8433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:09:58.942844 8433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:09:58.942876 8433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:09:58.942901 8433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:09:58.944262 8433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:45975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=32789
--tserver_master_addrs=127.27.117.126:38643,127.27.117.125:39543,127.27.117.124:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:09:58.944424 8433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:09:58.944591 8433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:09:58.949442 8445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.949462 8433 server_base.cc:1034] running on GCE node
W20260810 21:09:58.949561 8446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:09:58.949587 8452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:09:58.949954 8433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:09:58.950191 8433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:09:58.951367 8433 hybrid_clock.cc:648] HybridClock initialized: now 1786396198951343 us; error 35 us; skew 500 ppm
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:09:58.953644 8433 init.cc:377] Logged in from keytab as oryx/127.27.117.67@KRBTEST.COM (short username oryx)
I20260810 21:09:58.954087 8433 webserver.cc:460] Webserver started at http://127.27.117.67:32789/ using document root <none> and password file <none>
I20260810 21:09:58.954284 8433 fs_manager.cc:359] Metadata directory not provided
I20260810 21:09:58.954365 8433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:09:58.955658 8433 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:09:58.956207 8464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.956374 8433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:09:58.956470 8433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "f179a38fc7284926ae914ef8c268c16d"
format_stamp: "Formatted at 2026-08-10 21:09:57 on dist-test-slave-2kcd"
server_key: "04f8abfce72f922350021058fa3c1d06"
server_key_iv: "35c47d02b6cb6837e9cf197bfbf32b12"
server_key_version: "encryptionkey@0"
I20260810 21:09:58.956771 8433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:09:58.969352 8433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:09:58.970974 8433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:09:58.971149 8433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:09:58.971397 8433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:09:58.971694 8433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:09:58.971747 8433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.971781 8433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:09:58.971807 8433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:09:58.978331 8433 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:45975
I20260810 21:09:58.978384 8657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:45975 every 8 connection(s)
I20260810 21:09:58.978714 8433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.125@KRBTEST.COM
I20260810 21:09:58.985682 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 8433
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
Aug 10 21:09:58 dist-test-slave-2kcd krb5kdc[6378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396198, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.124@KRBTEST.COM
I20260810 21:09:58.989493 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.979404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.27.117.67:54719)
0810 21:09:58.979685 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.979696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.980649 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.980946 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.980947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.981276 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.981393 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.981395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.982090 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.982091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.983186 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.983188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.983617 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.983681 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.984232 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.984237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.985715 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.987786 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.987792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.987794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.988122 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.988346 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.988347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.988347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.988474 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.988825 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.988831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.988993 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.989094 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.989259 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":57,"spinlock_wait_cycles":6400,"thread_start_us":96,"threads_started":1}
I20260810 21:09:58.989815 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.979759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:54719 (local address 127.27.117.125:39543)
0810 21:09:58.979845 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.979846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.979855 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:58.980830 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.980831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.980844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.980891 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.981500 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.981977 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983267 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.983528 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983596 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.985822 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.985823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.985824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.985831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.987685 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.988213 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.988214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.988215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.988277 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.988552 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.988569 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.988569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.988707 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.988771 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.989637 (+ 866us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.989721 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 21:09:58.990339 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.979404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:53245)
0810 21:09:58.979825 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.979834 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.980649 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.980947 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.980948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.981273 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.981393 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.981395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.982329 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.982329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.982994 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.982996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.983439 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.983539 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.984068 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.984077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.986813 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.988817 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.988818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.988818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.989019 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.989236 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.989237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.989237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.989292 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.990048 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.990049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.990090 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.990145 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.990236 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":37,"thread_start_us":84,"threads_started":1}
I20260810 21:09:58.990399 8660 heartbeater.cc:347] Connected to a master server at 127.27.117.125:39543
I20260810 21:09:58.990483 8660 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.990669 8660 heartbeater.cc:511] Master 127.27.117.125:39543 requested a full tablet report, sending...
I20260810 21:09:58.990962 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.979694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53245 (local address 127.27.117.126:38643)
0810 21:09:58.979807 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.979809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.979885 (+ 76us) server_negotiation.cc:407] Connection header received
0810 21:09:58.980818 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.980820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.980833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.980884 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.981502 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.982242 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983100 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.983363 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983426 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.986920 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.986922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.986923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.986930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.988715 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.989106 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.989107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.989108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.989167 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.989359 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.989359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.989360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.989492 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.990737 (+ 1245us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.990780 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.990860 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 21:09:58.991074 7525 ts_manager.cc:194] Registered new tserver with Master: f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975)
I20260810 21:09:58.991592 8659 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38643
I20260810 21:09:58.991668 8659 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.991739 8659 heartbeater.cc:511] Master 127.27.117.126:38643 requested a full tablet report, sending...
I20260810 21:09:58.991964 6465 ts_manager.cc:194] Re-registered known tserver with Master: f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975)
I20260810 21:09:58.992419 6465 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:53245
I20260810 21:09:58.995244 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.979750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:42163 (local address 127.27.117.124:41067)
0810 21:09:58.979797 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.979800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.979811 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:09:58.980757 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.980758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.980773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.980847 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.981427 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.982202 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983647 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.983914 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983982 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.991937 (+ 7955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.991939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.991939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.991946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.994033 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.994445 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.994446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.994446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.994506 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.994719 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.994719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.994720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.994832 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.994870 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.995071 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.995152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:09:58.995815 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.979404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.27.117.67:42163)
0810 21:09:58.979581 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:58.979619 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:58.980629 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:58.980909 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:58.980917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:58.981167 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:09:58.981333 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.981337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.982282 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.982283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.983183 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:58.983185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.983978 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.983979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:58.984066 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.984405 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:09:58.984410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:09:58.991827 (+ 7417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:09:58.994118 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.994119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.994120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.994355 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.994593 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:09:58.994594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:09:58.994594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:09:58.994653 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:09:58.994910 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:09:58.994911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:09:58.994947 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 21:09:58.995559 (+ 612us) client_negotiation.cc:240] Negotiation successful
0810 21:09:58.995667 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260810 21:09:58.996296 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.986998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.27.117.125:39543)
0810 21:09:58.987851 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.987853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.987861 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:58.987880 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.987881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.987892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.988202 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.989095 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.989836 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.990931 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.991255 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.991586 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.992878 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.992879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.992880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.992886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.994786 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.995220 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.995221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.995221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.995285 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.995523 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.995524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.995524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.995643 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.996061 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.996104 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.996183 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":54,"threads_started":1}
I20260810 21:09:58.996678 8658 heartbeater.cc:347] Connected to a master server at 127.27.117.124:41067
I20260810 21:09:58.996766 8658 heartbeater.cc:464] Registering TS with master...
I20260810 21:09:58.996910 8658 heartbeater.cc:511] Master 127.27.117.124:41067 requested a full tablet report, sending...
I20260810 21:09:58.997267 7858 ts_manager.cc:194] Registered new tserver with Master: f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975)
I20260810 21:09:58.997871 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.987292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.27.117.126:38643)
0810 21:09:58.987396 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.987398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.987767 (+ 369us) server_negotiation.cc:407] Connection header received
0810 21:09:58.987874 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.987875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.987888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.987950 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.989385 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.990076 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.991185 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.991477 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.991543 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.994180 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.994181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.994182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.994206 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.996244 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.996834 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.996835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.996835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.996894 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.997132 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.997133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.997133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.997245 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.997353 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.997671 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.997771 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260810 21:09:58.999981 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:58.987425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.27.117.124:41067)
0810 21:09:58.987537 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:58.987539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:58.988318 (+ 779us) server_negotiation.cc:407] Connection header received
0810 21:09:58.988419 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:58.988421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:58.988435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:58.988477 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:58.992312 (+ 3835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.992979 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.995133 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:58.995462 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:58.995501 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:58.997004 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:58.997005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:58.997006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:58.997012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:58.998804 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.999209 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.999210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.999211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.999277 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:58.999480 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:58.999481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:58.999481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:58.999597 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:58.999697 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:58.999803 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 21:09:58.999883 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260810 21:09:59.000069 6465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:09:59.001740 6465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:09:59.002420 7796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: NON_VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: true } } }
I20260810 21:09:59.002532 7796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:59.003633 7796 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: attempting to promote NON_VOTER 12afdaf0442046bd9c39b48a2ecf9450 to VOTER
I20260810 21:09:59.003753 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } }
I20260810 21:09:59.004139 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Refusing update from remote peer a7587b80d5004d33880aa2b91613e611: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 21:09:59.004410 7878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 LEARNER]: Refusing update from remote peer a7587b80d5004d33880aa2b91613e611: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 21:09:59.004467 8681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 21:09:59.004846 8681 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 21:09:59.005479 7468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.005908 7878 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.006482 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.006557 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:59.006616 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.006664 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:59.006704 7468 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.006770 7468 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611 [sys.catalog]: This master's current role is: LEADER
I20260810 21:09:59.006963 7929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.007040 7929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:09:59.007644 7538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.008715 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7587b80d5004d33880aa2b91613e611" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:09:59.008916 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:09:59.010917 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.007219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45975 (local address 127.0.0.1:47156)
0810 21:09:59.007327 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:59.007338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:59.007379 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:59.007915 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:59.007916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:59.007918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:59.007998 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.008000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.009303 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.009304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.009977 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.009979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.010662 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.010663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.010720 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.010723 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:59.010774 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:09:59.010813 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 21:09:59.011253 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.007455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.27.117.67:45975)
0810 21:09:59.007745 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.007748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.007756 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:09:59.007782 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.007806 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.007820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.007895 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:59.008079 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.009224 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.010072 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.010531 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.010661 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.011037 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.011075 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.011117 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260810 21:09:59.011930 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 1c66d29bfc40471cbafd65d35cdd376c (DEFAULT_TABLE table=test-table [id=5b7d3b7d378349aebfa6645cabcd20f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:59.012233 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.007300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36243 (local address 127.0.0.1:34670)
0810 21:09:59.008004 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:59.008014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:59.008059 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:59.008461 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:59.008462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:59.008465 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:59.008507 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.008508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.009662 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.009663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.010901 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.010905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.012020 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.012021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.012095 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.012098 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:59.012129 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 21:09:59.012162 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":89,"threads_started":1}
I20260810 21:09:59.012331 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c66d29bfc40471cbafd65d35cdd376c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:59.012619 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.007397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.27.117.66:36243)
0810 21:09:59.008299 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.008303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.008313 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:09:59.008349 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.008351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.008365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.008443 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:59.008595 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.009575 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.011000 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.011934 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.012394 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.012479 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.012503 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.012541 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":103,"threads_started":1}
I20260810 21:09:59.013963 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.007495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36283 (local address 127.0.0.1:42918)
0810 21:09:59.008137 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:59.008145 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:59.008182 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:59.009910 (+ 1728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:59.009911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:59.009912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:59.009948 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.009949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.011838 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.011840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.012922 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.012925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.013814 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.013815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.013860 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.013863 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:59.013892 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 21:09:59.013919 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":89,"threads_started":1}
I20260810 21:09:59.014030 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.008804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.27.117.65:36283)
0810 21:09:59.009745 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.009749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.009758 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:59.009795 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.009797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.009812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.010015 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:59.010071 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.011741 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.013004 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.013729 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.013806 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.013869 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.013904 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.013941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":81,"threads_started":1}
I20260810 21:09:59.014654 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 1c66d29bfc40471cbafd65d35cdd376c (DEFAULT_TABLE table=test-table [id=5b7d3b7d378349aebfa6645cabcd20f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:59.014987 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c66d29bfc40471cbafd65d35cdd376c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:59.015511 8703 tablet_bootstrap.cc:492] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Bootstrap starting.
I20260810 21:09:59.016004 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 1c66d29bfc40471cbafd65d35cdd376c (DEFAULT_TABLE table=test-table [id=5b7d3b7d378349aebfa6645cabcd20f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:09:59.016238 8703 tablet_bootstrap.cc:654] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:59.016330 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c66d29bfc40471cbafd65d35cdd376c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:09:59.016582 8703 log.cc:824] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:59.017395 8703 tablet_bootstrap.cc:492] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: No bootstrap required, opened a new log
I20260810 21:09:59.017475 8703 ts_tablet_manager.cc:1397] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 21:09:59.018594 8709 tablet_bootstrap.cc:492] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Bootstrap starting.
I20260810 21:09:59.019292 8710 tablet_bootstrap.cc:492] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Bootstrap starting.
I20260810 21:09:59.019423 8709 tablet_bootstrap.cc:654] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:59.019556 8703 raft_consensus.cc:348] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.019726 8703 raft_consensus.cc:374] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:59.019706 8709 log.cc:824] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:59.019762 8703 raft_consensus.cc:729] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f179a38fc7284926ae914ef8c268c16d, State: Initialized, Role: FOLLOWER
I20260810 21:09:59.019923 8710 tablet_bootstrap.cc:654] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Neither blocks nor log segments found. Creating new log.
I20260810 21:09:59.019884 8703 consensus_queue.cc:260] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.020164 8703 ts_tablet_manager.cc:1428] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:09:59.020241 8710 log.cc:824] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Log is configured to *not* fsync() on all Append() calls
I20260810 21:09:59.020440 8709 tablet_bootstrap.cc:492] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: No bootstrap required, opened a new log
I20260810 21:09:59.020524 8709 ts_tablet_manager.cc:1397] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:09:59.020917 8659 heartbeater.cc:503] Master 127.27.117.126:38643 was elected leader, sending a full tablet report...
I20260810 21:09:59.021914 8710 tablet_bootstrap.cc:492] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: No bootstrap required, opened a new log
I20260810 21:09:59.021981 8710 ts_tablet_manager.cc:1397] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 21:09:59.022852 8709 raft_consensus.cc:348] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.022964 8709 raft_consensus.cc:374] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:59.022984 8709 raft_consensus.cc:729] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ecce09cf3248f2bddc714750d40946, State: Initialized, Role: FOLLOWER
I20260810 21:09:59.023082 8709 consensus_queue.cc:260] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.023288 8709 ts_tablet_manager.cc:1428] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:09:59.023375 8184 heartbeater.cc:503] Master 127.27.117.126:38643 was elected leader, sending a full tablet report...
I20260810 21:09:59.024492 8710 raft_consensus.cc:348] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.024605 8710 raft_consensus.cc:374] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:09:59.024624 8710 raft_consensus.cc:729] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0643f61c584a53a0eda4c92231ba22, State: Initialized, Role: FOLLOWER
I20260810 21:09:59.024750 8710 consensus_queue.cc:260] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.025045 8710 ts_tablet_manager.cc:1428] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 21:09:59.025107 8419 heartbeater.cc:503] Master 127.27.117.126:38643 was elected leader, sending a full tablet report...
W20260810 21:09:59.084347 8429 tablet.cc:2369] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:09:59.142081 8187 tablet.cc:2369] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:59.194653 8714 raft_consensus.cc:482] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:09:59.194766 8714 raft_consensus.cc:504] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.195140 8714 leader_election.cc:288] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66:36243), 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:09:59.199041 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.195529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:41077 (local address 127.27.117.65:36283)
0810 21:09:59.195604 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.195606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.195615 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:09:59.195641 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.195642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.195652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.195707 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:59.195841 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.196581 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.198123 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.198675 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.198758 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.198766 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.198929 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.198980 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 21:09:59.199040 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.195405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:43325 (local address 127.27.117.66:36243)
0810 21:09:59.195478 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.195480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.195509 (+ 29us) server_negotiation.cc:407] Connection header received
0810 21:09:59.195536 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.195537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.195546 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.195618 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:09:59.195829 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.196586 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.198127 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.198689 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.198757 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.198766 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.198934 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.198970 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:09:59.199054 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.195319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36243 (local address 127.27.117.67:43325)
0810 21:09:59.195394 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:59.195405 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:59.195473 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:59.195642 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:59.195643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:59.195644 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:59.195734 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.195736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.196664 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.196665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.197991 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.197994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.198773 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.198774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.198857 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.198860 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:09:59.198911 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:09:59.198955 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":101}
I20260810 21:09:59.199074 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.195443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36283 (local address 127.27.117.67:41077)
0810 21:09:59.195490 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:09:59.195501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:09:59.195540 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:09:59.195723 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:09:59.195723 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:09:59.195724 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:09:59.195774 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.195776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.196663 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.196665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.197991 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:09:59.197994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.198768 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.198769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:09:59.198846 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.198850 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:09:59.198911 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 21:09:59.198955 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":98}
I20260810 21:09:59.199579 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c66d29bfc40471cbafd65d35cdd376c" candidate_uuid: "f179a38fc7284926ae914ef8c268c16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff0643f61c584a53a0eda4c92231ba22" is_pre_election: true
I20260810 21:09:59.199568 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c66d29bfc40471cbafd65d35cdd376c" candidate_uuid: "f179a38fc7284926ae914ef8c268c16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ecce09cf3248f2bddc714750d40946" is_pre_election: true
I20260810 21:09:59.199750 8098 raft_consensus.cc:2454] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f179a38fc7284926ae914ef8c268c16d in term 0.
I20260810 21:09:59.199750 8340 raft_consensus.cc:2454] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f179a38fc7284926ae914ef8c268c16d in term 0.
I20260810 21:09:59.199986 8471 leader_election.cc:302] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ecce09cf3248f2bddc714750d40946, f179a38fc7284926ae914ef8c268c16d; no voters:
I20260810 21:09:59.200115 8714 raft_consensus.cc:2781] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:09:59.200174 8714 raft_consensus.cc:482] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:09:59.200198 8714 raft_consensus.cc:3037] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:59.201071 8714 raft_consensus.cc:504] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.201200 8714 leader_election.cc:288] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [CANDIDATE]: Term 1 election: Requested vote from peers ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66:36243), 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:09:59.201373 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c66d29bfc40471cbafd65d35cdd376c" candidate_uuid: "f179a38fc7284926ae914ef8c268c16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff0643f61c584a53a0eda4c92231ba22"
I20260810 21:09:59.201390 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c66d29bfc40471cbafd65d35cdd376c" candidate_uuid: "f179a38fc7284926ae914ef8c268c16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ecce09cf3248f2bddc714750d40946"
I20260810 21:09:59.201470 8098 raft_consensus.cc:3037] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:59.201464 8340 raft_consensus.cc:3037] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:09:59.202363 8340 raft_consensus.cc:2454] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f179a38fc7284926ae914ef8c268c16d in term 1.
I20260810 21:09:59.202363 8098 raft_consensus.cc:2454] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f179a38fc7284926ae914ef8c268c16d in term 1.
I20260810 21:09:59.202526 8471 leader_election.cc:302] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ecce09cf3248f2bddc714750d40946, f179a38fc7284926ae914ef8c268c16d; no voters:
I20260810 21:09:59.202618 8714 raft_consensus.cc:2781] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:09:59.202804 8714 raft_consensus.cc:686] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 1 LEADER]: Becoming Leader. State: Replica: f179a38fc7284926ae914ef8c268c16d, State: Running, Role: LEADER
I20260810 21:09:59.202909 8714 consensus_queue.cc:237] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } }
I20260810 21:09:59.203745 6465 catalog_manager.cc:5495] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d reported cstate change: term changed from 0 to 1, leader changed from <none> to f179a38fc7284926ae914ef8c268c16d (127.27.117.67). New cstate: current_term: 1 leader_uuid: "f179a38fc7284926ae914ef8c268c16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } health_report { overall_health: UNKNOWN } } }
W20260810 21:09:59.230227 8661 tablet.cc:2369] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:09:59.239356 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.237200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.27.117.67:45975)
0810 21:09:59.237266 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.237269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.237283 (+ 14us) server_negotiation.cc:407] Connection header received
0810 21:09:59.237312 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.237312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.237327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.237384 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:09:59.237507 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.237972 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.238682 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.238865 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.238913 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.239056 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:09:59.239146 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:09:59.239215 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.239252 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.239269 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 21:09:59.241063 8098 raft_consensus.cc:1261] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 1 FOLLOWER]: Refusing update from remote peer f179a38fc7284926ae914ef8c268c16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:59.241086 8340 raft_consensus.cc:1261] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Refusing update from remote peer f179a38fc7284926ae914ef8c268c16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:09:59.241467 8714 consensus_queue.cc:1035] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:59.241597 8722 consensus_queue.cc:1035] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:09:59.244632 8728 mvcc.cc:205] Tried to move back new op lower bound from 7317078832088596480 to 7317078831935713280. Current Snapshot: MvccSnapshot[applied={T|T < 7317078832088596480}]
I20260810 21:09:59.244884 8727 mvcc.cc:205] Tried to move back new op lower bound from 7317078832088596480 to 7317078831935713280. Current Snapshot: MvccSnapshot[applied={T|T < 7317078832088596480}]
I20260810 21:09:59.247886 6465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:59.247978 6465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:09:59.249264 6465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7587b80d5004d33880aa2b91613e611: Sending DeleteTablet for 3 replicas of tablet 1c66d29bfc40471cbafd65d35cdd376c
I20260810 21:09:59.249631 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c66d29bfc40471cbafd65d35cdd376c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:59 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47156
I20260810 21:09:59.249794 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c66d29bfc40471cbafd65d35cdd376c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:59 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:42918
I20260810 21:09:59.250087 8741 tablet_replica.cc:330] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: stopping tablet replica
I20260810 21:09:59.250227 8741 raft_consensus.cc:2229] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:59.250345 8741 raft_consensus.cc:2258] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:59.250818 8741 ts_tablet_manager.cc:1905] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:59.251294 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 6403
I20260810 21:09:59.252552 8741 ts_tablet_manager.cc:1918] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:59.252629 8741 log.cc:1196] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/1c66d29bfc40471cbafd65d35cdd376c
I20260810 21:09:59.252950 8741 ts_tablet_manager.cc:1939] T 1c66d29bfc40471cbafd65d35cdd376c P 48ecce09cf3248f2bddc714750d40946: Deleting consensus metadata
I20260810 21:09:59.261480 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c66d29bfc40471cbafd65d35cdd376c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:09:59 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:34670
I20260810 21:09:59.262409 8744 tablet_replica.cc:330] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: stopping tablet replica
I20260810 21:09:59.262524 8744 raft_consensus.cc:2229] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:09:59.262650 8744 raft_consensus.cc:2258] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:59.262634 8740 tablet_replica.cc:330] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: stopping tablet replica
I20260810 21:09:59.263015 8740 raft_consensus.cc:2229] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:09:59.263139 8744 ts_tablet_manager.cc:1905] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:59.263276 8740 raft_consensus.cc:2258] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:09:59.263845 8740 ts_tablet_manager.cc:1905] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:09:59.264917 8744 ts_tablet_manager.cc:1918] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:59.265008 8744 log.cc:1196] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/1c66d29bfc40471cbafd65d35cdd376c
I20260810 21:09:59.265305 8744 ts_tablet_manager.cc:1939] T 1c66d29bfc40471cbafd65d35cdd376c P ff0643f61c584a53a0eda4c92231ba22: Deleting consensus metadata
I20260810 21:09:59.266260 8740 ts_tablet_manager.cc:1918] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:09:59.266345 8740 log.cc:1196] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/1c66d29bfc40471cbafd65d35cdd376c
I20260810 21:09:59.266613 8740 ts_tablet_manager.cc:1939] T 1c66d29bfc40471cbafd65d35cdd376c P f179a38fc7284926ae914ef8c268c16d: Deleting consensus metadata
W20260810 21:09:59.269150 7978 connection.cc:535] server connection from 127.0.0.1:42918 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 21:09:59.269160 8237 connection.cc:535] server connection from 127.0.0.1:34670 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 21:09:59.269286 8471 connection.cc:535] server connection from 127.0.0.1:47156 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 21:09:59.278640 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.271151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.27.117.125:39543)
0810 21:09:59.271196 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.271198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.271323 (+ 125us) server_negotiation.cc:407] Connection header received
0810 21:09:59.271361 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.271362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.271376 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.271437 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:59.272168 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.272707 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.273663 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.274001 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.274076 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.275376 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:59.275378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:59.275378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:59.275385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:59.277336 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:59.277730 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:59.277731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:59.277732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:59.277795 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:59.278055 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:59.278056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:59.278057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:59.278201 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:59.278311 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.278433 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.278525 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:09:59.279281 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:09:59.271061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.27.117.124:41067)
0810 21:09:59.271106 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:09:59.271108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:09:59.271356 (+ 248us) server_negotiation.cc:407] Connection header received
0810 21:09:59.271446 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:09:59.271447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:09:59.271461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:09:59.271507 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:09:59.272271 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.273065 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.273923 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:09:59.274201 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:09:59.274278 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:09:59.275966 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:09:59.275967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:09:59.275968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:09:59.275976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:09:59.277909 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:59.278305 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:59.278306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:59.278306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:59.278361 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:09:59.278591 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:09:59.278592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:09:59.278593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:09:59.278822 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:09:59.278897 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 21:09:59.279095 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 21:09:59.279179 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:10:00.246703 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.246405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:49883)
0810 21:10:00.246553 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.246643 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260810 21:10:00.246706 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.246398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:36165)
0810 21:10:00.246565 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.246649 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260810 21:10:00.246925 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.246716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:60015)
0810 21:10:00.246821 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.246886 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
W20260810 21:10:00.246961 8419 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:38643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
W20260810 21:10:00.247074 8184 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:38643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
W20260810 21:10:00.247208 8659 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:38643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
I20260810 21:10:00.749938 8766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a7587b80d5004d33880aa2b91613e611)
I20260810 21:10:00.750137 8766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } }
I20260810 21:10:00.750475 8766 leader_election.cc:288] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643), f0752cc294db4e56b0e12f0dfce8e7cb (127.27.117.125:39543)
I20260810 21:10:00.750813 8769 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a7587b80d5004d33880aa2b91613e611)
I20260810 21:10:00.750900 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.750653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51700)
0810 21:10:00.750767 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.750853 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
W20260810 21:10:00.751035 7830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
I20260810 21:10:00.751133 8769 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } }
I20260810 21:10:00.751477 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.751356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51716)
0810 21:10:00.751409 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.751443 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 21:10:00.751565 7830 leader_election.cc:334] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643): Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
I20260810 21:10:00.751756 8769 leader_election.cc:288] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643), 12afdaf0442046bd9c39b48a2ecf9450 (127.27.117.124:41067)
I20260810 21:10:00.751724 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.751482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51728)
0810 21:10:00.751643 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.751689 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":59,"threads_started":1}
W20260810 21:10:00.752051 7503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
I20260810 21:10:00.752532 8778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.752439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51752)
0810 21:10:00.752477 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.752508 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 21:10:00.752671 7503 leader_election.cc:334] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643): Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
I20260810 21:10:00.754222 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.750690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39543 (local address 127.0.0.1:43998)
0810 21:10:00.750806 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.750817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.750878 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.751054 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.751056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.751059 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.751122 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.751123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.752444 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.752445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.753384 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.753387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.753971 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.753972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.754035 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.754039 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.754112 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.754156 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":47,"threads_started":1}
I20260810 21:10:00.754220 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.750769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.27.117.125:39543)
0810 21:10:00.750877 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.750879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.750890 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:10:00.750947 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.750949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.750960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.751026 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.751244 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.752361 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.753496 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.753875 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.753967 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.753977 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.754116 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.754157 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260810 21:10:00.754503 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12afdaf0442046bd9c39b48a2ecf9450" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" is_pre_election: true
I20260810 21:10:00.754611 7538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12afdaf0442046bd9c39b48a2ecf9450 in term 1.
I20260810 21:10:00.754751 7827 leader_election.cc:302] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12afdaf0442046bd9c39b48a2ecf9450, f0752cc294db4e56b0e12f0dfce8e7cb; no voters: a7587b80d5004d33880aa2b91613e611
I20260810 21:10:00.754884 8766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 21:10:00.755010 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.751541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41067 (local address 127.0.0.1:45876)
0810 21:10:00.751794 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.751804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.751873 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.752405 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.752406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.752409 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.752458 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.752460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.753252 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.753253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.754011 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.754014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.754745 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.754746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.754809 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.754813 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.754891 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.754937 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260810 21:10:00.754961 8766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a7587b80d5004d33880aa2b91613e611)
I20260810 21:10:00.755039 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.751762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.27.117.124:41067)
0810 21:10:00.752268 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.752270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.752278 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:00.752298 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.752299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.752308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.752387 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.752544 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.753164 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.754104 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.754642 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.754744 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.754753 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.754916 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.754948 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":101,"threads_started":1}
I20260810 21:10:00.755084 8766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 1 FOLLOWER]: Advancing to term 2
I20260810 21:10:00.755303 7878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12afdaf0442046bd9c39b48a2ecf9450" is_pre_election: true
I20260810 21:10:00.756012 8766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } }
I20260810 21:10:00.756147 7878 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f0752cc294db4e56b0e12f0dfce8e7cb in current term 2: Already voted for candidate 12afdaf0442046bd9c39b48a2ecf9450 in this term.
I20260810 21:10:00.756165 8766 leader_election.cc:288] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [CANDIDATE]: Term 2 election: Requested vote from peers a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643), f0752cc294db4e56b0e12f0dfce8e7cb (127.27.117.125:39543)
I20260810 21:10:00.756300 7538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12afdaf0442046bd9c39b48a2ecf9450" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb"
I20260810 21:10:00.756383 7538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 1 FOLLOWER]: Advancing to term 2
I20260810 21:10:00.756409 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.756234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51756)
0810 21:10:00.756334 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.756364 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 21:10:00.756729 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.756634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51760)
0810 21:10:00.756680 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.756706 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 21:10:00.756372 7504 leader_election.cc:302] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f0752cc294db4e56b0e12f0dfce8e7cb; no voters: 12afdaf0442046bd9c39b48a2ecf9450, a7587b80d5004d33880aa2b91613e611
W20260810 21:10:00.756829 7830 leader_election.cc:334] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643): Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
I20260810 21:10:00.757244 7538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12afdaf0442046bd9c39b48a2ecf9450 in term 2.
I20260810 21:10:00.757359 8769 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:10:00.757421 7827 leader_election.cc:302] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12afdaf0442046bd9c39b48a2ecf9450, f0752cc294db4e56b0e12f0dfce8e7cb; no voters: a7587b80d5004d33880aa2b91613e611
I20260810 21:10:00.757525 8766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 21:10:00.757689 8766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [term 2 LEADER]: Becoming Leader. State: Replica: 12afdaf0442046bd9c39b48a2ecf9450, State: Running, Role: LEADER
I20260810 21:10:00.757772 8766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } }
I20260810 21:10:00.758118 8782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12afdaf0442046bd9c39b48a2ecf9450. Latest consensus state: current_term: 2 leader_uuid: "12afdaf0442046bd9c39b48a2ecf9450" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:10:00.758195 8782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:00.758374 8786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:00.758890 8786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:10:00.758994 8786 catalog_manager.cc:1259] Loaded cluster ID: e3f5693b3d654f8498d63913b10dcb90
I20260810 21:10:00.759047 8786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:10:00.759208 8786 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:00.759307 8786 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Loaded TSK: 0
I20260810 21:10:00.759461 8786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:10:00.809101 7850 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:10:00.810767 7850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:00.811826 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.811692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51790)
0810 21:10:00.811751 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.811787 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 21:10:00.811834 7538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [term 2 FOLLOWER]: Refusing update from remote peer 12afdaf0442046bd9c39b48a2ecf9450: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 21:10:00.812235 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.812086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.0.0.1:51804)
0810 21:10:00.812137 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.812167 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 21:10:00.812170 8766 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 21:10:00.812327 7830 consensus_peers.cc:487] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 -> Peer a7587b80d5004d33880aa2b91613e611 (127.27.117.126:38643): Couldn't send request to peer a7587b80d5004d33880aa2b91613e611. Status: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 21:10:00.813006 8769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12afdaf0442046bd9c39b48a2ecf9450. Latest consensus state: current_term: 2 leader_uuid: "12afdaf0442046bd9c39b48a2ecf9450" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:10:00.813135 8769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:10:00.813484 8788 mvcc.cc:205] Tried to move back new op lower bound from 7317078838523072512 to 7317078838304493568. Current Snapshot: MvccSnapshot[applied={T|T < 7317078838523072512}]
I20260810 21:10:00.813581 8782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "12afdaf0442046bd9c39b48a2ecf9450" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:10:00.813675 8782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:00.813827 8769 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "12afdaf0442046bd9c39b48a2ecf9450" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a7587b80d5004d33880aa2b91613e611" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38643 } } peers { permanent_uuid: "f0752cc294db4e56b0e12f0dfce8e7cb" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39543 } attrs { promote: false } } peers { permanent_uuid: "12afdaf0442046bd9c39b48a2ecf9450" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41067 } attrs { promote: false } } }
I20260810 21:10:00.813926 8769 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0752cc294db4e56b0e12f0dfce8e7cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:10:00.819154 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.815562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45975 (local address 127.0.0.1:42576)
0810 21:10:00.815728 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.815740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.815799 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.816143 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.816144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.816146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.816214 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.816215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.816957 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.816958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.818124 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.818126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.818899 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.818961 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.818965 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.819037 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.819086 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":46}
I20260810 21:10:00.819185 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.815624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.27.117.67:45975)
0810 21:10:00.815993 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.815995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.816004 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:00.816030 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.816030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.816053 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.816182 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.816308 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.816860 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818212 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.818806 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818890 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.818907 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.819046 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.819107 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":65,"threads_started":1}
I20260810 21:10:00.819299 8806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.815793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.27.117.66:36243)
0810 21:10:00.815950 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.815952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.815961 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:00.815993 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.815994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.816005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.816236 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.816300 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.817074 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818289 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.818856 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818953 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.818965 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.819181 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.819228 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260810 21:10:00.819658 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.815719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36243 (local address 127.0.0.1:43502)
0810 21:10:00.815863 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.815873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.815901 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.816105 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.816106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.816108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.816164 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.816166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.817156 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.817157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.818204 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.818206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.818965 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.819040 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.819044 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.819094 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.819136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260810 21:10:00.819944 8307 tablet_service.cc:1467] Processing CreateTablet for tablet 60086def00414cd6beabcd70e918cab5 (DEFAULT_TABLE table=test-table [id=cce7ce1d215f4b9babbe034cb3cd9ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:00.819977 8537 tablet_service.cc:1467] Processing CreateTablet for tablet 60086def00414cd6beabcd70e918cab5 (DEFAULT_TABLE table=test-table [id=cce7ce1d215f4b9babbe034cb3cd9ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:00.820111 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60086def00414cd6beabcd70e918cab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:00.820123 8537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60086def00414cd6beabcd70e918cab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:00.820272 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.815675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.27.117.65:36283)
0810 21:10:00.815778 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.815780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.815914 (+ 134us) server_negotiation.cc:407] Connection header received
0810 21:10:00.816019 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.816020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.816032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.816075 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.816299 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.817082 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.819125 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.819759 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.819805 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.819817 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.820135 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.820180 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
I20260810 21:10:00.820470 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.815621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36283 (local address 127.0.0.1:46472)
0810 21:10:00.815857 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.815867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.815937 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.816147 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.816148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.816149 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.816214 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.816215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.818292 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.818293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.819019 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.819021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.819976 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.819976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.820044 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.820048 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.820376 (+ 328us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.820418 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1}
I20260810 21:10:00.820679 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 60086def00414cd6beabcd70e918cab5 (DEFAULT_TABLE table=test-table [id=cce7ce1d215f4b9babbe034cb3cd9ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:00.820825 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60086def00414cd6beabcd70e918cab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:00.821532 8807 tablet_bootstrap.cc:492] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Bootstrap starting.
I20260810 21:10:00.821708 8809 tablet_bootstrap.cc:492] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: Bootstrap starting.
I20260810 21:10:00.821996 8807 tablet_bootstrap.cc:654] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:00.822163 8809 tablet_bootstrap.cc:654] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:00.822383 8815 tablet_bootstrap.cc:492] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946: Bootstrap starting.
I20260810 21:10:00.822654 8807 tablet_bootstrap.cc:492] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: No bootstrap required, opened a new log
I20260810 21:10:00.822698 8807 ts_tablet_manager.cc:1397] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:00.822818 8807 raft_consensus.cc:348] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.822898 8807 raft_consensus.cc:374] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:00.822913 8807 raft_consensus.cc:729] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f179a38fc7284926ae914ef8c268c16d, State: Initialized, Role: FOLLOWER
I20260810 21:10:00.822996 8809 tablet_bootstrap.cc:492] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: No bootstrap required, opened a new log
I20260810 21:10:00.823004 8807 consensus_queue.cc:260] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.823077 8809 ts_tablet_manager.cc:1397] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:00.823140 8807 ts_tablet_manager.cc:1428] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:00.823468 8815 tablet_bootstrap.cc:654] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:00.823560 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.823363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:45615)
0810 21:10:00.823494 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.823528 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260810 21:10:00.823792 8658 heartbeater.cc:503] Master 127.27.117.124:41067 was elected leader, sending a full tablet report...
I20260810 21:10:00.823882 8809 raft_consensus.cc:348] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.824051 8809 raft_consensus.cc:374] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:00.824090 8815 tablet_bootstrap.cc:492] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946: No bootstrap required, opened a new log
I20260810 21:10:00.824152 8815 ts_tablet_manager.cc:1397] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:10:00.824131 8809 raft_consensus.cc:729] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0643f61c584a53a0eda4c92231ba22, State: Initialized, Role: FOLLOWER
I20260810 21:10:00.824276 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.823905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:48821)
0810 21:10:00.824218 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.824248 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299}
I20260810 21:10:00.824316 8815 raft_consensus.cc:348] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.824394 8815 raft_consensus.cc:374] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:00.824416 8815 raft_consensus.cc:729] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ecce09cf3248f2bddc714750d40946, State: Initialized, Role: FOLLOWER
I20260810 21:10:00.824405 8809 consensus_queue.cc:260] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.824467 8815 consensus_queue.cc:260] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.824612 8815 ts_tablet_manager.cc:1428] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:00.824609 8809 ts_tablet_manager.cc:1428] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:00.825038 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.824802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:35237)
0810 21:10:00.824961 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.824994 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260810 21:10:00.825060 8416 heartbeater.cc:503] Master 127.27.117.124:41067 was elected leader, sending a full tablet report...
I20260810 21:10:00.825405 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.825234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:56995)
0810 21:10:00.825353 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.825380 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260810 21:10:00.825592 8182 heartbeater.cc:503] Master 127.27.117.124:41067 was elected leader, sending a full tablet report...
I20260810 21:10:00.825701 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.825230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:53665)
0810 21:10:00.825584 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.825620 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":99,"threads_started":1}
I20260810 21:10:00.826041 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.825930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:39899)
0810 21:10:00.825990 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.826013 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 21:10:00.984182 8821 raft_consensus.cc:482] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:00.984288 8821 raft_consensus.cc:504] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.984710 8821 leader_election.cc:288] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975), 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:10:00.987865 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.985052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:45681 (local address 127.27.117.65:36283)
0810 21:10:00.985099 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.985101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.985112 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:10:00.985142 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.985143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.985153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.985217 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.985433 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.986192 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.987161 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.987522 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.987605 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.987616 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.987759 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.987798 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:10:00.987874 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.985057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:45909 (local address 127.27.117.67:45975)
0810 21:10:00.985099 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.985101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.985112 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:10:00.985143 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.985144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.985153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.985217 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:00.985432 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.986198 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.987144 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.987516 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.987612 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.987622 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.987769 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.987810 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:10:00.987880 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.984961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36283 (local address 127.27.117.66:45681)
0810 21:10:00.985011 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.985022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.985070 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.985235 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.985236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.985237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.985351 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.985352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.986310 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.986312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.987054 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.987056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.987605 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.987606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.987681 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.987683 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.987753 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.987791 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":25}
I20260810 21:10:00.987890 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.984915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45975 (local address 127.27.117.66:45909)
0810 21:10:00.984987 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.985000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:00.985070 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:00.985235 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:00.985236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:00.985237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:00.985351 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.985352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.986280 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.986281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.987033 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:00.987047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.987613 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.987614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:00.987690 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.987694 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:00.987761 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 21:10:00.987800 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":6}
I20260810 21:10:00.988180 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60086def00414cd6beabcd70e918cab5" candidate_uuid: "ff0643f61c584a53a0eda4c92231ba22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ecce09cf3248f2bddc714750d40946" is_pre_election: true
I20260810 21:10:00.988274 8098 raft_consensus.cc:2454] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff0643f61c584a53a0eda4c92231ba22 in term 0.
I20260810 21:10:00.988279 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60086def00414cd6beabcd70e918cab5" candidate_uuid: "ff0643f61c584a53a0eda4c92231ba22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f179a38fc7284926ae914ef8c268c16d" is_pre_election: true
I20260810 21:10:00.988415 8570 raft_consensus.cc:2454] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff0643f61c584a53a0eda4c92231ba22 in term 0.
I20260810 21:10:00.988487 8237 leader_election.cc:302] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48ecce09cf3248f2bddc714750d40946, ff0643f61c584a53a0eda4c92231ba22; no voters:
I20260810 21:10:00.988658 8821 raft_consensus.cc:2781] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:00.988720 8821 raft_consensus.cc:482] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:00.988746 8821 raft_consensus.cc:3037] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:00.989454 8821 raft_consensus.cc:504] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.989606 8821 leader_election.cc:288] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [CANDIDATE]: Term 1 election: Requested vote from peers f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975), 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283)
I20260810 21:10:00.989748 8570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60086def00414cd6beabcd70e918cab5" candidate_uuid: "ff0643f61c584a53a0eda4c92231ba22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f179a38fc7284926ae914ef8c268c16d"
I20260810 21:10:00.989779 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60086def00414cd6beabcd70e918cab5" candidate_uuid: "ff0643f61c584a53a0eda4c92231ba22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48ecce09cf3248f2bddc714750d40946"
I20260810 21:10:00.989842 8570 raft_consensus.cc:3037] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:00.989851 8098 raft_consensus.cc:3037] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:00.990485 8570 raft_consensus.cc:2454] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff0643f61c584a53a0eda4c92231ba22 in term 1.
I20260810 21:10:00.990504 8098 raft_consensus.cc:2454] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff0643f61c584a53a0eda4c92231ba22 in term 1.
I20260810 21:10:00.990692 8237 leader_election.cc:302] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48ecce09cf3248f2bddc714750d40946, f179a38fc7284926ae914ef8c268c16d, ff0643f61c584a53a0eda4c92231ba22; no voters:
I20260810 21:10:00.990783 8821 raft_consensus.cc:2781] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:00.990922 8821 raft_consensus.cc:686] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 1 LEADER]: Becoming Leader. State: Replica: ff0643f61c584a53a0eda4c92231ba22, State: Running, Role: LEADER
I20260810 21:10:00.991150 8821 consensus_queue.cc:237] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } }
I20260810 21:10:00.991221 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.991145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:50281)
0810 21:10:00.991173 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:00.991202 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260810 21:10:00.991932 7850 catalog_manager.cc:5495] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff0643f61c584a53a0eda4c92231ba22 (127.27.117.66). New cstate: current_term: 1 leader_uuid: "ff0643f61c584a53a0eda4c92231ba22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff0643f61c584a53a0eda4c92231ba22" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36243 } health_report { overall_health: HEALTHY } } }
I20260810 21:10:00.998564 8806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:00.996575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.27.117.66:36243)
0810 21:10:00.996614 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:00.996615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:00.996629 (+ 14us) server_negotiation.cc:407] Connection header received
0810 21:10:00.996653 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:00.996654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:00.996667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:00.996724 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:00.996879 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.997302 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.997978 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:00.998148 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:00.998197 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:00.998334 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:00.998417 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:00.998470 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:00.998506 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 21:10:00.998523 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:10:00.999467 8098 raft_consensus.cc:1261] T 60086def00414cd6beabcd70e918cab5 P 48ecce09cf3248f2bddc714750d40946 [term 1 FOLLOWER]: Refusing update from remote peer ff0643f61c584a53a0eda4c92231ba22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:00.999581 8570 raft_consensus.cc:1261] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Refusing update from remote peer ff0643f61c584a53a0eda4c92231ba22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:00.999857 8827 consensus_queue.cc:1035] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f179a38fc7284926ae914ef8c268c16d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:01.000005 8821 consensus_queue.cc:1035] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ecce09cf3248f2bddc714750d40946" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:01.001219 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:01.001041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:50075)
0810 21:10:01.001161 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.001188 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260810 21:10:01.001574 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:01.001439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:58867)
0810 21:10:01.001520 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.001546 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 21:10:01.002672 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:01.002583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.66:58451)
0810 21:10:01.002616 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.002647 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 21:10:01.002689 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:01.002595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.65:43981)
0810 21:10:01.002629 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.002657 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 21:10:01.002992 8835 mvcc.cc:205] Tried to move back new op lower bound from 7317078839292256256 to 7317078839260590080. Current Snapshot: MvccSnapshot[applied={T|T < 7317078839292256256}]
I20260810 21:10:01.003096 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:01.003017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38643 (local address 127.27.117.67:39331)
0810 21:10:01.003052 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.003076 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:38643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 21:10:01.005247 7850 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:01.005335 7850 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:01.006495 7850 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12afdaf0442046bd9c39b48a2ecf9450: Sending DeleteTablet for 3 replicas of tablet 60086def00414cd6beabcd70e918cab5
I20260810 21:10:01.006765 8537 tablet_service.cc:1514] Processing DeleteTablet for tablet 60086def00414cd6beabcd70e918cab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:01 UTC) from {username='oryx', principal='oryx/127.27.117.124@KRBTEST.COM'} at 127.0.0.1:42576
I20260810 21:10:01.006923 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 60086def00414cd6beabcd70e918cab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:01 UTC) from {username='oryx', principal='oryx/127.27.117.124@KRBTEST.COM'} at 127.0.0.1:46472
I20260810 21:10:01.007012 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 7959
I20260810 21:10:01.007169 8307 tablet_service.cc:1514] Processing DeleteTablet for tablet 60086def00414cd6beabcd70e918cab5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:01 UTC) from {username='oryx', principal='oryx/127.27.117.124@KRBTEST.COM'} at 127.0.0.1:43502
I20260810 21:10:01.014416 8850 tablet_replica.cc:330] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: stopping tablet replica
I20260810 21:10:01.014477 8850 raft_consensus.cc:2229] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:01.014520 8850 raft_consensus.cc:2258] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:01.014901 8850 ts_tablet_manager.cc:1905] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:01.016683 8850 ts_tablet_manager.cc:1918] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:01.016738 8850 log.cc:1196] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/60086def00414cd6beabcd70e918cab5
I20260810 21:10:01.017040 8850 ts_tablet_manager.cc:1939] T 60086def00414cd6beabcd70e918cab5 P f179a38fc7284926ae914ef8c268c16d: Deleting consensus metadata
I20260810 21:10:01.017403 7830 catalog_manager.cc:4841] TS f179a38fc7284926ae914ef8c268c16d (127.27.117.67:45975): tablet 60086def00414cd6beabcd70e918cab5 (table test-table [id=cce7ce1d215f4b9babbe034cb3cd9ea4]) successfully deleted
I20260810 21:10:01.021359 8858 tablet_replica.cc:330] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: stopping tablet replica
I20260810 21:10:01.021436 8858 raft_consensus.cc:2229] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:10:01.021562 8858 raft_consensus.cc:2258] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:01.021847 8858 ts_tablet_manager.cc:1905] T 60086def00414cd6beabcd70e918cab5 P ff0643f61c584a53a0eda4c92231ba22: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 21:10:01.022302 8803 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:01.021740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36283 (local address 127.0.0.1:46488)
0810 21:10:01.021785 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.021796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:01.021833 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:01.022255 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:36283: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 21:10:01.022450 7831 catalog_manager.cc:4584] TS 48ecce09cf3248f2bddc714750d40946 (127.27.117.65:36283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60086def00414cd6beabcd70e918cab5: Network error: Client connection negotiation failed: client connection to 127.27.117.65:36283: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 21:10:01.022927 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 8196
I20260810 21:10:01.038951 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:01.038831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36243 (local address 127.0.0.1:43514)
0810 21:10:01.038890 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:01.038921 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.66:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 21:10:01.039098 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 8433
I20260810 21:10:01.051786 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 7292
I20260810 21:10:01.061291 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 7616
2026-08-10T21:10:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5179 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:10:01 dist-test-slave-2kcd krb5kdc[8862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:10:01 dist-test-slave-2kcd krb5kdc[8862](info): set up 2 sockets
Aug 10 21:10:01 dist-test-slave-2kcd krb5kdc[8862](info): commencing operation
krb5kdc: starting...
W20260810 21:10:02.335819 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab.
Entry for principal oryx/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab.
I20260810 21:10:02.453576 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:37361
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab
--principal=oryx/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:02.535048 8890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:02.535254 8890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:02.535279 8890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:02.536441 8890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:02.536485 8890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 21:10:02.536499 8890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:02.536512 8890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:02.536576 8890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:02.536594 8890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:02.538211 8890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:37361
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.126.keytab
--principal=oryx/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:02.538462 8890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:02.538704 8890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:02.543565 8901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:02.543701 8890 server_base.cc:1034] running on GCE node
W20260810 21:10:02.543565 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:02.543828 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:02.544162 8890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:02.544464 8890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:02.545604 8890 hybrid_clock.cc:648] HybridClock initialized: now 1786396202545597 us; error 36 us; skew 500 ppm
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:02.552297 8890 init.cc:377] Logged in from keytab as oryx/127.27.117.126@KRBTEST.COM (short username oryx)
I20260810 21:10:02.552717 8890 webserver.cc:460] Webserver started at http://127.27.117.126:44723/ using document root <none> and password file <none>
I20260810 21:10:02.552968 8890 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:02.553015 8890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:02.553172 8890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:02.554248 8890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "f96dceb453e4415a9a911e44120fe299"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "8e38aaf42c55ab708fed825b34482c9e"
server_key_iv: "5cdbb94527ab082806de383984480489"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.554615 8890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "f96dceb453e4415a9a911e44120fe299"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "8e38aaf42c55ab708fed825b34482c9e"
server_key_iv: "5cdbb94527ab082806de383984480489"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.555917 8890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:10:02.556627 8918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:02.556839 8890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:02.556950 8890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "f96dceb453e4415a9a911e44120fe299"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "8e38aaf42c55ab708fed825b34482c9e"
server_key_iv: "5cdbb94527ab082806de383984480489"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.557044 8890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:02.569201 8890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:02.571069 8890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:02.571231 8890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:02.576253 8890 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:37361
I20260810 21:10:02.576303 9024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:37361 every 8 connection(s)
I20260810 21:10:02.576648 8890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:10:02.577376 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:02.578030 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 8890
I20260810 21:10:02.578131 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:10:02.578352 28117 external_mini_cluster.cc:1428] Setting key a41280de067f815aa5c7a8711e6206b4
I20260810 21:10:02.580681 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Bootstrap starting.
I20260810 21:10:02.581367 9025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:02.581705 9025 log.cc:824] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:02.582459 9025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: No bootstrap required, opened a new log
I20260810 21:10:02.585147 9025 raft_consensus.cc:348] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96dceb453e4415a9a911e44120fe299" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37361 } }
I20260810 21:10:02.585282 9025 raft_consensus.cc:374] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:02.585304 9025 raft_consensus.cc:729] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96dceb453e4415a9a911e44120fe299, State: Initialized, Role: FOLLOWER
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:02.585445 9025 consensus_queue.cc:260] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96dceb453e4415a9a911e44120fe299" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37361 } }
I20260810 21:10:02.585517 9025 raft_consensus.cc:388] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:10:02.585541 9025 raft_consensus.cc:482] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:10:02.585609 9025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:02.586198 9025 raft_consensus.cc:504] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96dceb453e4415a9a911e44120fe299" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37361 } }
I20260810 21:10:02.586287 9025 leader_election.cc:302] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96dceb453e4415a9a911e44120fe299; no voters:
I20260810 21:10:02.586501 9025 leader_election.cc:288] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:10:02.586547 9033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:02.586680 9033 raft_consensus.cc:686] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [term 1 LEADER]: Becoming Leader. State: Replica: f96dceb453e4415a9a911e44120fe299, State: Running, Role: LEADER
I20260810 21:10:02.586763 9033 consensus_queue.cc:237] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96dceb453e4415a9a911e44120fe299" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37361 } }
I20260810 21:10:02.586769 9025 sys_catalog.cc:564] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:10:02.587318 9035 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96dceb453e4415a9a911e44120fe299. Latest consensus state: current_term: 1 leader_uuid: "f96dceb453e4415a9a911e44120fe299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96dceb453e4415a9a911e44120fe299" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37361 } } }
I20260810 21:10:02.587415 9035 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:02.587759 9047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:02.587894 9036 sys_catalog.cc:455] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96dceb453e4415a9a911e44120fe299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96dceb453e4415a9a911e44120fe299" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37361 } } }
I20260810 21:10:02.588044 9036 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:02.588315 9047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:10:02.589879 9047 catalog_manager.cc:1347] Generated new cluster ID: 5c31fee45d754e149d32ae6f1f6f8c0d
I20260810 21:10:02.589932 9047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:10:02.589980 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.578941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.27.117.126:37361)
0810 21:10:02.579101 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:02.579110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:02.579142 (+ 32us) server_negotiation.cc:407] Connection header received
0810 21:10:02.579631 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:02.579637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:02.579683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:02.579798 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:02.580587 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.581564 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.582572 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.582982 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.583065 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.585926 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:02.585930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:02.585930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:02.585940 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:02.588274 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.588698 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.588702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.588702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.588848 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.589064 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.589064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.589065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.589344 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:02.589538 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:02.589611 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 21:10:02.589748 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260810 21:10:02.609942 9047 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:10:02.610618 9047 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:02.617532 9047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Generated new TSK 0
I20260810 21:10:02.617709 9047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab.
Entry for principal oryx/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab.
I20260810 21:10:02.651589 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:37361
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:02.747118 9078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:02.747313 9078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:02.747336 9078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:02.748495 9078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:02.748538 9078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:02.748574 9078 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:10:02.748593 9078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:10:02.750213 9078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:02.750456 9078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:02.750694 9078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:02.755688 9094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:02.755662 9091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:02.755650 9092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:02.755998 9078 server_base.cc:1034] running on GCE node
I20260810 21:10:02.756222 9078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:02.756493 9078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:02.757671 9078 hybrid_clock.cc:648] HybridClock initialized: now 1786396202757652 us; error 50 us; skew 500 ppm
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:02.760685 9078 init.cc:377] Logged in from keytab as oryx/127.27.117.65@KRBTEST.COM (short username oryx)
I20260810 21:10:02.761112 9078 webserver.cc:460] Webserver started at http://127.27.117.65:44757/ using document root <none> and password file <none>
I20260810 21:10:02.761372 9078 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:02.761442 9078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:02.761585 9078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:02.762735 9078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "5ea8f946143741398434dbd52fe9f3ab"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "ad60341ae01b94d45e0358b568bea74e"
server_key_iv: "0f0e4a599246678916088659fe924318"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.763110 9078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea8f946143741398434dbd52fe9f3ab"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "ad60341ae01b94d45e0358b568bea74e"
server_key_iv: "0f0e4a599246678916088659fe924318"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.764647 9078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:10:02.765640 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:02.765834 9078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:02.765930 9078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "5ea8f946143741398434dbd52fe9f3ab"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "ad60341ae01b94d45e0358b568bea74e"
server_key_iv: "0f0e4a599246678916088659fe924318"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.766009 9078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:02.777544 9078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:02.779788 9078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:02.779933 9078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:02.780403 9078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:02.780467 9078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:02.780506 9078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:02.780527 9078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:02.788300 9078 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:46565
I20260810 21:10:02.788364 9313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:46565 every 8 connection(s)
I20260810 21:10:02.788724 9078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:10:02.792079 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.780953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:43569 (local address 127.27.117.126:37361)
0810 21:10:02.781001 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:02.781004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:02.781225 (+ 221us) server_negotiation.cc:407] Connection header received
0810 21:10:02.781975 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:02.781976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:02.781992 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:02.782075 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:02.782912 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.783730 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.785348 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.785725 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.785817 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.788824 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:02.788826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:02.788826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:02.788835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:02.790439 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.791033 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.791034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.791034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.791115 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.791431 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.791432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.791432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.791531 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:02.791583 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:02.791941 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 21:10:02.792007 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:10:02.792369 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.780911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.27.117.65:43569)
0810 21:10:02.781072 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:02.781119 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:02.781800 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:02.782147 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:02.782151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:02.782499 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:02.782759 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.782764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.783823 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.783824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.785261 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.785265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.785837 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.785838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.786011 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.786568 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:02.786581 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:02.788700 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:02.790527 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.790535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.790538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.790873 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.791188 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.791189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.791190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.791353 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.791603 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:02.791610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:02.791785 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 21:10:02.791925 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 21:10:02.792117 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260810 21:10:02.792946 9314 heartbeater.cc:347] Connected to a master server at 127.27.117.126:37361
I20260810 21:10:02.793059 9314 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:02.793239 9314 heartbeater.cc:511] Master 127.27.117.126:37361 requested a full tablet report, sending...
I20260810 21:10:02.793792 8946 ts_manager.cc:194] Registered new tserver with Master: 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565)
I20260810 21:10:02.794580 8946 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:43569
I20260810 21:10:02.797722 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 9078
I20260810 21:10:02.797840 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:10:02.798074 28117 external_mini_cluster.cc:1428] Setting key 874a1e30ca31befe7429729f42948d64
I20260810 21:10:02.802845 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.794110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.27.117.65:32989)
0810 21:10:02.794272 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:02.794284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:02.794351 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:02.794482 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:02.794483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:02.794705 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:02.794772 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.794774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.795452 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.795453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.796450 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.796454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.796822 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.796823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.796905 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.797413 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:02.797419 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:02.797860 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:02.801668 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.801669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.801669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.801905 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.802134 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.802135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.802136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.802204 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.802478 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:02.802479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:02.802526 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 21:10:02.802596 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 21:10:02.802704 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":35,"thread_start_us":80,"threads_started":1}
I20260810 21:10:02.802845 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.794170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:32989 (local address 127.27.117.126:37361)
0810 21:10:02.794213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:02.794215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:02.794349 (+ 134us) server_negotiation.cc:407] Connection header received
0810 21:10:02.794410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:02.794411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:02.794421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:02.794477 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:02.794875 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.795345 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.796547 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.796720 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.796776 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.797979 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:02.797981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:02.797981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:02.797989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:02.801578 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.801989 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.801990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.801990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.802061 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.802273 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.802274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.802275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.802409 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:02.802477 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:02.802637 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 21:10:02.802720 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab.
Entry for principal oryx/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab.
I20260810 21:10:02.825672 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:37361
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:02.926476 9331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:02.926764 9331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:02.926802 9331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:02.928046 9331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:02.928087 9331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:02.928108 9331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:10:02.928151 9331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:10:02.930498 9331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:02.930773 9331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:02.931066 9331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:02.938124 9340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:02.938134 9347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:02.938136 9339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:02.938431 9331 server_base.cc:1034] running on GCE node
I20260810 21:10:02.938797 9331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:02.939077 9331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:02.940243 9331 hybrid_clock.cc:648] HybridClock initialized: now 1786396202940231 us; error 32 us; skew 500 ppm
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:02.942912 9331 init.cc:377] Logged in from keytab as oryx/127.27.117.66@KRBTEST.COM (short username oryx)
I20260810 21:10:02.943377 9331 webserver.cc:460] Webserver started at http://127.27.117.66:37797/ using document root <none> and password file <none>
I20260810 21:10:02.943612 9331 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:02.943679 9331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:02.943818 9331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:02.945039 9331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "33157240609e4adcac666cd24499be77"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "b1a13fdb8cb5f9440cb0beff32c5775c"
server_key_iv: "d2f613dc82ceec0b7f9d03acef7aa36a"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.945413 9331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "33157240609e4adcac666cd24499be77"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "b1a13fdb8cb5f9440cb0beff32c5775c"
server_key_iv: "d2f613dc82ceec0b7f9d03acef7aa36a"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.946853 9331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:02.947664 9363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:02.947866 9331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:02.947999 9331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "33157240609e4adcac666cd24499be77"
format_stamp: "Formatted at 2026-08-10 21:10:02 on dist-test-slave-2kcd"
server_key: "b1a13fdb8cb5f9440cb0beff32c5775c"
server_key_iv: "d2f613dc82ceec0b7f9d03acef7aa36a"
server_key_version: "encryptionkey@0"
I20260810 21:10:02.948071 9331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:02.958693 9331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:02.960727 9331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:02.960902 9331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:02.961404 9331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:02.961467 9331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:02.961515 9331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:02.961547 9331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:10:02 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:02.969372 9331 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:43515
I20260810 21:10:02.969658 9590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:43515 every 8 connection(s)
I20260810 21:10:02.969717 9331 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:10:02.971885 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.961930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:60345 (local address 127.27.117.126:37361)
0810 21:10:02.961995 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:02.961997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:02.962193 (+ 196us) server_negotiation.cc:407] Connection header received
0810 21:10:02.962715 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:02.962717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:02.962729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:02.962798 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:02.963499 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.963995 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.965196 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.965457 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.965532 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.969087 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:02.969088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:02.969089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:02.969097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:02.970688 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.971107 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.971109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.971109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.971176 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.971377 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.971378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.971378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.971484 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:02.971536 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:02.971761 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 21:10:02.971822 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:10:02.972085 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.961921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.27.117.66:60345)
0810 21:10:02.962077 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:02.962111 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:02.962609 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:02.962862 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:02.962866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:02.963176 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:02.963405 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.963410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.964101 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.964102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.965106 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.965108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.965562 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.965563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.965712 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.966229 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:02.966239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:02.968972 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:02.970760 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.970765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.970766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.970980 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.971227 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.971227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.971228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.971322 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.971542 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:02.971546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:02.971654 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 21:10:02.971736 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 21:10:02.971855 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260810 21:10:02.972513 9591 heartbeater.cc:347] Connected to a master server at 127.27.117.126:37361
I20260810 21:10:02.972594 9591 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:02.972774 9591 heartbeater.cc:511] Master 127.27.117.126:37361 requested a full tablet report, sending...
I20260810 21:10:02.973171 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 9331
I20260810 21:10:02.973232 8947 ts_manager.cc:194] Registered new tserver with Master: 33157240609e4adcac666cd24499be77 (127.27.117.66:43515)
I20260810 21:10:02.973280 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:10:02.973460 28117 external_mini_cluster.cc:1428] Setting key 9b8b15f1a69fd36e269a94d518ef5d76
I20260810 21:10:02.973671 8947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:60345
I20260810 21:10:02.979568 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.974017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:44139 (local address 127.27.117.126:37361)
0810 21:10:02.974115 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:02.974117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:02.974185 (+ 68us) server_negotiation.cc:407] Connection header received
0810 21:10:02.974257 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:02.974258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:02.974268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:02.974323 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:02.974640 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.975097 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.975786 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.975984 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.976068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.977022 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:02.977024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:02.977024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:02.977031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:02.978385 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.978744 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.978745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.978745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.978805 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:02.979018 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:02.979019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:02.979019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:02.979168 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:02.979242 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:02.979399 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 21:10:02.979480 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 21:10:02.979575 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:02.973973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.27.117.66:44139)
0810 21:10:02.974107 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:02.974117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:02.974199 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:02.974328 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:02.974329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:02.974509 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:02.974557 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.974558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.975194 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.975195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.975703 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:02.975705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:02.976071 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:02.976073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:02.976143 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:02.976567 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:02.976572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:02.976916 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:02.978499 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.978500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.978501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.978671 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.978878 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:02.978879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:02.978879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:02.978943 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:02.979245 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:02.979246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:02.979291 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 21:10:02.979374 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 21:10:02.979476 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab.
Entry for principal oryx/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab.
I20260810 21:10:03.000021 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:37361
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:03.082185 9607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:03.082376 9607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:03.082398 9607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:03.083454 9607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:03.083487 9607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:03.083506 9607 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 21:10:03.083524 9607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:10:03.085058 9607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.117.126:37361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:03.085227 9607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:03.085386 9607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:03.090432 9621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:03.090426 9623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:03.090590 9607 server_base.cc:1034] running on GCE node
W20260810 21:10:03.090693 9620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:03.090904 9607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:03.091176 9607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:03.092339 9607 hybrid_clock.cc:648] HybridClock initialized: now 1786396203092323 us; error 29 us; skew 500 ppm
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396203, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:03.094867 9607 init.cc:377] Logged in from keytab as oryx/127.27.117.67@KRBTEST.COM (short username oryx)
I20260810 21:10:03.095314 9607 webserver.cc:460] Webserver started at http://127.27.117.67:40983/ using document root <none> and password file <none>
I20260810 21:10:03.095531 9607 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:03.095602 9607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:03.095767 9607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:03.096765 9607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "6d7e866667bf4fb4a295b31cb703451e"
format_stamp: "Formatted at 2026-08-10 21:10:03 on dist-test-slave-2kcd"
server_key: "c2e6d5fdd13ccb2da32ce5572dd58048"
server_key_iv: "5821fcaa3606ba234db5d5d792ce15a0"
server_key_version: "encryptionkey@0"
I20260810 21:10:03.097200 9607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "6d7e866667bf4fb4a295b31cb703451e"
format_stamp: "Formatted at 2026-08-10 21:10:03 on dist-test-slave-2kcd"
server_key: "c2e6d5fdd13ccb2da32ce5572dd58048"
server_key_iv: "5821fcaa3606ba234db5d5d792ce15a0"
server_key_version: "encryptionkey@0"
I20260810 21:10:03.098529 9607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:10:03.099193 9635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.099367 9607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:10:03.099473 9607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "6d7e866667bf4fb4a295b31cb703451e"
format_stamp: "Formatted at 2026-08-10 21:10:03 on dist-test-slave-2kcd"
server_key: "c2e6d5fdd13ccb2da32ce5572dd58048"
server_key_iv: "5821fcaa3606ba234db5d5d792ce15a0"
server_key_version: "encryptionkey@0"
I20260810 21:10:03.099551 9607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:03.110857 9607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:03.112525 9607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:03.112684 9607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:03.113267 9607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:03.113329 9607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.113368 9607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:03.113396 9607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396203, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:03.120729 9607 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:36767
I20260810 21:10:03.120819 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:36767 every 8 connection(s)
I20260810 21:10:03.121093 9607 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:10:03.122988 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.113617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53331 (local address 127.27.117.126:37361)
0810 21:10:03.113677 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.113679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.113905 (+ 226us) server_negotiation.cc:407] Connection header received
0810 21:10:03.114316 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.114318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.114333 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.114417 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.115050 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.115895 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.116956 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.117185 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.117272 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.119914 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:03.119916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:03.119916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:03.119924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:03.121633 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.122020 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.122021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.122022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.122079 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.122333 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.122334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.122334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.122450 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:03.122528 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.122811 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.122903 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:10:03.123128 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.113615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.27.117.67:53331)
0810 21:10:03.113769 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.113807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.114223 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.114460 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.114463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.114748 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:03.114952 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.114962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.115995 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.115996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.116870 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.116872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.117283 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.117284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.117414 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.117904 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:03.117929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:03.119821 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:03.121720 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:03.121726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:03.121728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:03.121936 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:03.122154 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:03.122155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:03.122155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:03.122260 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:03.122532 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:03.122539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:03.122665 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.122767 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.122912 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260810 21:10:03.123549 9842 heartbeater.cc:347] Connected to a master server at 127.27.117.126:37361
I20260810 21:10:03.123632 9842 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:03.123816 9842 heartbeater.cc:511] Master 127.27.117.126:37361 requested a full tablet report, sending...
I20260810 21:10:03.124225 8947 ts_manager.cc:194] Registered new tserver with Master: 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.124696 8947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:53331
I20260810 21:10:03.124959 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 9607
I20260810 21:10:03.125079 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:10:03.125289 28117 external_mini_cluster.cc:1428] Setting key e8ccffd7fb16e1078906cf7d07ffaa62
I20260810 21:10:03.126149 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:10:03.130892 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.124583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:47415 (local address 127.27.117.126:37361)
0810 21:10:03.124620 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.124622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.124666 (+ 44us) server_negotiation.cc:407] Connection header received
0810 21:10:03.124731 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.124732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.124739 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.124844 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.125287 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.125875 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.126852 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.127033 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.127084 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.127997 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:03.127999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:03.127999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:03.128004 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:03.129686 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.130169 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.130170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.130170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.130239 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.130473 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.130474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.130474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.130569 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:03.130622 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.130781 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.130838 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:10:03.131026 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.124482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.27.117.67:47415)
0810 21:10:03.124613 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.124623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.124679 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.124893 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.124894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.125122 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:03.125199 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.125201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.125991 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.125992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.126769 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.126772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.127106 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.127108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.127174 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.127535 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:03.127540 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:03.127913 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:03.129792 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:03.129792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:03.129793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:03.129950 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:03.130326 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:03.130327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:03.130327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:03.130403 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:03.130635 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:03.130636 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:03.130686 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.130760 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.130870 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 21:10:03.150008 9031 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:03.144630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.27.117.126:37361)
0810 21:10:03.144695 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.144698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.144708 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:10:03.144739 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.144740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.144748 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.144820 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.145386 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.145891 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.146657 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.146847 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.146931 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.149897 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35222: BlockingRecv error: recv got EOF from 127.0.0.1:35222 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:03.157106 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.150803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.27.117.126:37361)
0810 21:10:03.150846 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.150848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.150927 (+ 79us) server_negotiation.cc:407] Connection header received
0810 21:10:03.150964 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.150965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.150973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.151020 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.151465 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.151923 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.152596 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.152764 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.152860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.154752 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:03.154753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:03.154753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:03.154759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:03.156102 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.156405 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.156406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.156407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.156464 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.156610 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.156610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.156610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.156699 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:03.156770 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.156965 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.157024 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:10:03.158717 8947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:10:03.160339 8947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:03.167073 9874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.162582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43515 (local address 127.0.0.1:47586)
0810 21:10:03.162921 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.162966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.163033 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.163296 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.163299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.163301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.163369 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.163370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.164648 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.164649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.165875 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.165877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.166748 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.166827 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.166841 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.166918 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.166988 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":146,"thread_start_us":68,"threads_started":1}
I20260810 21:10:03.167119 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.162744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.27.117.66:43515)
0810 21:10:03.162984 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.162988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.163015 (+ 27us) server_negotiation.cc:407] Connection header received
0810 21:10:03.163165 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.163167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.163198 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.163277 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.163436 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.164554 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.165963 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.166650 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166730 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.166805 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.166948 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.166998 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
I20260810 21:10:03.167275 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.162895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.27.117.67:36767)
0810 21:10:03.163093 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.163096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.163512 (+ 416us) server_negotiation.cc:407] Connection header received
0810 21:10:03.163642 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.163645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.163664 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.163743 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.163861 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.164541 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166186 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.166678 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166772 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.166849 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.167118 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.167167 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260810 21:10:03.167474 9883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.162799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:36767 (local address 127.0.0.1:43700)
0810 21:10:03.163343 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.163352 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.163541 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.163741 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.163742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.163744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.163806 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.163807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.164630 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.164631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.166079 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.166082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.166882 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.166951 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.166955 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.167001 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.167033 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":88,"thread_start_us":66,"threads_started":1}
I20260810 21:10:03.167649 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.162708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.27.117.65:46565)
0810 21:10:03.163103 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.163106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.163248 (+ 142us) server_negotiation.cc:407] Connection header received
0810 21:10:03.163330 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.163333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.163438 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.163520 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.163763 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.164767 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166200 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.166932 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.166999 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.167074 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.167487 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.167537 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":110,"threads_started":1}
I20260810 21:10:03.167650 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.162583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:46565 (local address 127.0.0.1:60930)
0810 21:10:03.163150 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.163159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.163258 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.163634 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.163635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.163637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.163681 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.163682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.164901 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.164902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.166120 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.166122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.167112 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.167113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.167380 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.167386 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.167449 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.167490 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":5,"spinlock_wait_cycles":384,"thread_start_us":96,"threads_started":1}
I20260810 21:10:03.167878 9455 tablet_service.cc:1467] Processing CreateTablet for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c (DEFAULT_TABLE table=test-table [id=02b8808b8e334c46b0dde9a1304ccba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.168256 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.169454 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c (DEFAULT_TABLE table=test-table [id=02b8808b8e334c46b0dde9a1304ccba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.169492 9710 tablet_service.cc:1467] Processing CreateTablet for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c (DEFAULT_TABLE table=test-table [id=02b8808b8e334c46b0dde9a1304ccba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.169831 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.169829 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.171813 9885 tablet_bootstrap.cc:492] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Bootstrap starting.
I20260810 21:10:03.172456 9885 tablet_bootstrap.cc:654] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.172837 9885 log.cc:824] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:03.173635 9885 tablet_bootstrap.cc:492] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: No bootstrap required, opened a new log
I20260810 21:10:03.173664 9893 tablet_bootstrap.cc:492] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Bootstrap starting.
I20260810 21:10:03.173712 9885 ts_tablet_manager.cc:1397] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:10:03.173978 9894 tablet_bootstrap.cc:492] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Bootstrap starting.
I20260810 21:10:03.174283 9893 tablet_bootstrap.cc:654] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.174595 9894 tablet_bootstrap.cc:654] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.174604 9893 log.cc:824] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:03.174964 9894 log.cc:824] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:03.175462 9893 tablet_bootstrap.cc:492] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: No bootstrap required, opened a new log
I20260810 21:10:03.175580 9893 ts_tablet_manager.cc:1397] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:10:03.175848 9894 tablet_bootstrap.cc:492] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: No bootstrap required, opened a new log
I20260810 21:10:03.175901 9894 ts_tablet_manager.cc:1397] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:10:03.176031 9885 raft_consensus.cc:348] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.176224 9885 raft_consensus.cc:374] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.176260 9885 raft_consensus.cc:729] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33157240609e4adcac666cd24499be77, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.176399 9885 consensus_queue.cc:260] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.176671 9885 ts_tablet_manager.cc:1428] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:10:03.176842 9591 heartbeater.cc:503] Master 127.27.117.126:37361 was elected leader, sending a full tablet report...
I20260810 21:10:03.177771 9893 raft_consensus.cc:348] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.177891 9893 raft_consensus.cc:374] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.177912 9893 raft_consensus.cc:729] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.177857 9894 raft_consensus.cc:348] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.178035 9894 raft_consensus.cc:374] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.178015 9893 consensus_queue.cc:260] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.178093 9894 raft_consensus.cc:729] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea8f946143741398434dbd52fe9f3ab, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.178217 9894 consensus_queue.cc:260] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.178269 9893 ts_tablet_manager.cc:1428] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 21:10:03.178289 9842 heartbeater.cc:503] Master 127.27.117.126:37361 was elected leader, sending a full tablet report...
I20260810 21:10:03.178483 9894 ts_tablet_manager.cc:1428] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:10:03.178596 9314 heartbeater.cc:503] Master 127.27.117.126:37361 was elected leader, sending a full tablet report...
W20260810 21:10:03.220425 9592 tablet.cc:2369] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:10:03.289564 9315 tablet.cc:2369] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:10:03.371757 9843 tablet.cc:2369] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:10:03.415192 9899 raft_consensus.cc:482] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:03.415334 9899 raft_consensus.cc:504] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.415714 9899 leader_election.cc:288] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33157240609e4adcac666cd24499be77 (127.27.117.66:43515), 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565)
I20260810 21:10:03.418586 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.415983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:47113 (local address 127.27.117.66:43515)
0810 21:10:03.416028 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.416030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.416068 (+ 38us) server_negotiation.cc:407] Connection header received
0810 21:10:03.416139 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.416140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.416150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.416224 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.416387 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.416931 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.417832 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.418195 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.418278 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.418288 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.418473 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.418510 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:10:03.418603 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.415929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43515 (local address 127.27.117.67:47113)
0810 21:10:03.416002 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.416014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.416078 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.416228 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.416229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.416230 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.416309 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.416311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.417012 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.417013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.417732 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.417734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.418275 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.418276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.418370 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.418377 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.418466 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.418508 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 21:10:03.418664 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.416061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:33463 (local address 127.27.117.65:46565)
0810 21:10:03.416107 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.416109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.416262 (+ 153us) server_negotiation.cc:407] Connection header received
0810 21:10:03.416337 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.416338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.416348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.416415 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.416585 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.417062 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.417937 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.418291 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.418376 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.418384 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.418553 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.418593 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:10:03.418700 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.416012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:46565 (local address 127.27.117.67:33463)
0810 21:10:03.416185 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.416193 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.416272 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.416419 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.416420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.416421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.416512 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.416514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.417145 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.417146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.417845 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.417847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.418377 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.418378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.418476 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.418479 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.418547 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.418587 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260810 21:10:03.418949 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9179a1f1386b4d9d8a35a3b6c5321d1c" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33157240609e4adcac666cd24499be77" is_pre_election: true
I20260810 21:10:03.418990 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9179a1f1386b4d9d8a35a3b6c5321d1c" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab" is_pre_election: true
I20260810 21:10:03.419124 9505 raft_consensus.cc:2454] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 0.
I20260810 21:10:03.419150 9224 raft_consensus.cc:2454] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 0.
I20260810 21:10:03.419380 9642 leader_election.cc:302] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:03.419513 9899 raft_consensus.cc:2781] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:03.419570 9899 raft_consensus.cc:482] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:03.419608 9899 raft_consensus.cc:3037] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.420609 9899 raft_consensus.cc:504] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.420754 9899 leader_election.cc:288] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 election: Requested vote from peers 33157240609e4adcac666cd24499be77 (127.27.117.66:43515), 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565)
I20260810 21:10:03.420919 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9179a1f1386b4d9d8a35a3b6c5321d1c" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab"
I20260810 21:10:03.420921 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9179a1f1386b4d9d8a35a3b6c5321d1c" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33157240609e4adcac666cd24499be77"
I20260810 21:10:03.421000 9224 raft_consensus.cc:3037] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.421021 9505 raft_consensus.cc:3037] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.421690 9224 raft_consensus.cc:2454] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 1.
I20260810 21:10:03.421721 9505 raft_consensus.cc:2454] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 1.
I20260810 21:10:03.421911 9641 leader_election.cc:302] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea8f946143741398434dbd52fe9f3ab, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:03.422024 9899 raft_consensus.cc:2781] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:03.422155 9899 raft_consensus.cc:686] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 1 LEADER]: Becoming Leader. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Running, Role: LEADER
I20260810 21:10:03.422272 9899 consensus_queue.cc:237] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.423086 8946 catalog_manager.cc:5495] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67). New cstate: current_term: 1 leader_uuid: "6d7e866667bf4fb4a295b31cb703451e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } health_report { overall_health: HEALTHY } } }
I20260810 21:10:03.453415 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.451135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.27.117.67:36767)
0810 21:10:03.451200 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.451202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.451213 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:10:03.451251 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.451252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.451271 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.451334 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:03.451544 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.452011 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.452744 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.452947 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.452998 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.453152 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.453246 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.453307 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.453342 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.453359 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:10:03.455380 9224 raft_consensus.cc:1261] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Refusing update from remote peer 6d7e866667bf4fb4a295b31cb703451e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.455380 9505 raft_consensus.cc:1261] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Refusing update from remote peer 6d7e866667bf4fb4a295b31cb703451e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.455755 9903 consensus_queue.cc:1035] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.455901 9899 consensus_queue.cc:1035] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.458568 9908 mvcc.cc:205] Tried to move back new op lower bound from 7317078849349955584 to 7317078849218248704. Current Snapshot: MvccSnapshot[applied={T|T < 7317078849349955584}]
I20260810 21:10:03.458844 9907 mvcc.cc:205] Tried to move back new op lower bound from 7317078849349955584 to 7317078849218248704. Current Snapshot: MvccSnapshot[applied={T|T < 7317078849349955584}]
I20260810 21:10:03.460804 8946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:03.460894 8946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:03.461695 8946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Sending DeleteTablet for 3 replicas of tablet 9179a1f1386b4d9d8a35a3b6c5321d1c
I20260810 21:10:03.462023 9455 tablet_service.cc:1514] Processing DeleteTablet for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47586
I20260810 21:10:03.462033 9710 tablet_service.cc:1514] Processing DeleteTablet for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43700
I20260810 21:10:03.462250 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 9179a1f1386b4d9d8a35a3b6c5321d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60930
I20260810 21:10:03.462366 9928 tablet_replica.cc:330] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: stopping tablet replica
I20260810 21:10:03.462446 9927 tablet_replica.cc:330] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: stopping tablet replica
I20260810 21:10:03.462539 9928 raft_consensus.cc:2229] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:10:03.462553 9927 raft_consensus.cc:2229] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:03.462646 9927 raft_consensus.cc:2258] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.462752 9928 raft_consensus.cc:2258] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.462739 8946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:10:03.462882 8946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:03.463143 9930 tablet_replica.cc:330] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: stopping tablet replica
I20260810 21:10:03.463255 9930 raft_consensus.cc:2229] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:03.463354 9930 raft_consensus.cc:2258] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.463405 9928 ts_tablet_manager.cc:1905] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.463807 9930 ts_tablet_manager.cc:1905] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.464636 9190 tablet_service.cc:1467] Processing CreateTablet for tablet c2d32dce4468444f8cb71944d2981836 (DEFAULT_TABLE table=test-table [id=b32622020a554cb8a5245f55c92f7bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.464892 9455 tablet_service.cc:1467] Processing CreateTablet for tablet c2d32dce4468444f8cb71944d2981836 (DEFAULT_TABLE table=test-table [id=b32622020a554cb8a5245f55c92f7bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.465016 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d32dce4468444f8cb71944d2981836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.465209 9928 ts_tablet_manager.cc:1918] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:03.465279 9928 log.cc:1196] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/9179a1f1386b4d9d8a35a3b6c5321d1c
I20260810 21:10:03.465560 9928 ts_tablet_manager.cc:1939] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 6d7e866667bf4fb4a295b31cb703451e: Deleting consensus metadata
I20260810 21:10:03.465977 8925 catalog_manager.cc:4841] TS 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767): tablet 9179a1f1386b4d9d8a35a3b6c5321d1c (table test-table [id=02b8808b8e334c46b0dde9a1304ccba7]) successfully deleted
I20260810 21:10:03.466346 9885 tablet_bootstrap.cc:492] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Bootstrap starting.
I20260810 21:10:03.466408 9710 tablet_service.cc:1467] Processing CreateTablet for tablet c2d32dce4468444f8cb71944d2981836 (DEFAULT_TABLE table=test-table [id=b32622020a554cb8a5245f55c92f7bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.466512 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d32dce4468444f8cb71944d2981836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.466790 9885 tablet_bootstrap.cc:654] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.467434 9885 tablet_bootstrap.cc:492] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: No bootstrap required, opened a new log
I20260810 21:10:03.467463 9893 tablet_bootstrap.cc:492] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Bootstrap starting.
I20260810 21:10:03.467495 9885 ts_tablet_manager.cc:1397] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.467639 9885 raft_consensus.cc:348] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.467697 9885 raft_consensus.cc:374] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.467718 9885 raft_consensus.cc:729] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33157240609e4adcac666cd24499be77, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.467762 9885 consensus_queue.cc:260] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.467842 9885 ts_tablet_manager.cc:1428] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.467892 9893 tablet_bootstrap.cc:654] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.468528 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d32dce4468444f8cb71944d2981836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.468602 9893 tablet_bootstrap.cc:492] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: No bootstrap required, opened a new log
I20260810 21:10:03.468652 9893 ts_tablet_manager.cc:1397] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.468813 9893 raft_consensus.cc:348] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.468878 9893 raft_consensus.cc:374] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.468904 9893 raft_consensus.cc:729] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.468945 9893 consensus_queue.cc:260] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.469022 9893 ts_tablet_manager.cc:1428] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:03.469890 9930 ts_tablet_manager.cc:1918] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:03.469899 9894 tablet_bootstrap.cc:492] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Bootstrap starting.
I20260810 21:10:03.469983 9930 log.cc:1196] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/9179a1f1386b4d9d8a35a3b6c5321d1c
I20260810 21:10:03.470258 9930 ts_tablet_manager.cc:1939] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 5ea8f946143741398434dbd52fe9f3ab: Deleting consensus metadata
I20260810 21:10:03.470347 9894 tablet_bootstrap.cc:654] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.470686 8924 catalog_manager.cc:4841] TS 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565): tablet 9179a1f1386b4d9d8a35a3b6c5321d1c (table test-table [id=02b8808b8e334c46b0dde9a1304ccba7]) successfully deleted
I20260810 21:10:03.471052 9927 ts_tablet_manager.cc:1905] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.472270 9927 ts_tablet_manager.cc:1918] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:03.472378 9927 log.cc:1196] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/9179a1f1386b4d9d8a35a3b6c5321d1c
I20260810 21:10:03.472433 9894 tablet_bootstrap.cc:492] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: No bootstrap required, opened a new log
I20260810 21:10:03.472487 9894 ts_tablet_manager.cc:1397] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 21:10:03.472657 9927 ts_tablet_manager.cc:1939] T 9179a1f1386b4d9d8a35a3b6c5321d1c P 33157240609e4adcac666cd24499be77: Deleting consensus metadata
I20260810 21:10:03.472883 9894 raft_consensus.cc:348] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.472973 9894 raft_consensus.cc:374] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.472997 9894 raft_consensus.cc:729] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea8f946143741398434dbd52fe9f3ab, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.473059 8925 catalog_manager.cc:4841] TS 33157240609e4adcac666cd24499be77 (127.27.117.66:43515): tablet 9179a1f1386b4d9d8a35a3b6c5321d1c (table test-table [id=02b8808b8e334c46b0dde9a1304ccba7]) successfully deleted
I20260810 21:10:03.473052 9894 consensus_queue.cc:260] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.473179 9894 ts_tablet_manager.cc:1428] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:10:03.541996 9897 raft_consensus.cc:482] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:03.542109 9897 raft_consensus.cc:504] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.542500 9897 leader_election.cc:288] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.546164 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.542809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:49131 (local address 127.27.117.67:36767)
0810 21:10:03.542857 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.542859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.542985 (+ 126us) server_negotiation.cc:407] Connection header received
0810 21:10:03.543095 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.543097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.543109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.543168 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.543311 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.543781 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.545280 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.545824 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.545906 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.545915 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.546059 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.546100 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:10:03.546167 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.542665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:36767 (local address 127.27.117.66:49131)
0810 21:10:03.542918 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.542927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.543014 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.543185 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.543186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.543187 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.543250 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.543251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.543869 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.543870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.545186 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.545189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.545910 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.545911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.545968 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.545971 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.546043 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.546091 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":62,"thread_start_us":62,"threads_started":1}
I20260810 21:10:03.546809 9748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d32dce4468444f8cb71944d2981836" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7e866667bf4fb4a295b31cb703451e" is_pre_election: true
I20260810 21:10:03.546881 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.542665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:46565 (local address 127.27.117.66:35761)
0810 21:10:03.542800 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.542813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.542876 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.543105 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.543106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.543108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:03.543171 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.543174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.544116 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.544117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.545201 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.545203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.546619 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.546621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.546689 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.546692 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.546766 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.546802 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260810 21:10:03.546916 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.542800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:35761 (local address 127.27.117.65:46565)
0810 21:10:03.542850 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.542852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.542864 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:10:03.543009 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.543011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.543021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.543087 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:03.543276 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.544018 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.545954 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.546504 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.546576 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.546584 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.546802 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.546853 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:10:03.546940 9748 raft_consensus.cc:2454] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 0.
I20260810 21:10:03.547224 9373 leader_election.cc:302] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:03.547266 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d32dce4468444f8cb71944d2981836" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab" is_pre_election: true
I20260810 21:10:03.547353 9897 raft_consensus.cc:2781] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:03.547387 9897 raft_consensus.cc:482] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:03.547384 9224 raft_consensus.cc:2454] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 0.
I20260810 21:10:03.547413 9897 raft_consensus.cc:3037] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.548142 9897 raft_consensus.cc:504] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.548290 9897 leader_election.cc:288] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.548499 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d32dce4468444f8cb71944d2981836" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab"
I20260810 21:10:03.548524 9748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d32dce4468444f8cb71944d2981836" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7e866667bf4fb4a295b31cb703451e"
I20260810 21:10:03.548593 9224 raft_consensus.cc:3037] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.548628 9748 raft_consensus.cc:3037] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.549398 9748 raft_consensus.cc:2454] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 1.
I20260810 21:10:03.549417 9224 raft_consensus.cc:2454] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 1.
I20260810 21:10:03.549604 9372 leader_election.cc:302] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 5ea8f946143741398434dbd52fe9f3ab; no voters:
I20260810 21:10:03.549705 9897 raft_consensus.cc:2781] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:03.549826 9897 raft_consensus.cc:686] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 1 LEADER]: Becoming Leader. State: Replica: 33157240609e4adcac666cd24499be77, State: Running, Role: LEADER
I20260810 21:10:03.549896 9897 consensus_queue.cc:237] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.550719 8946 catalog_manager.cc:5495] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33157240609e4adcac666cd24499be77 (127.27.117.66). New cstate: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } health_report { overall_health: UNKNOWN } } }
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396202, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.126@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:03.592295 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.585655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.27.117.126:37361)
0810 21:10:03.585738 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.585740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.585861 (+ 121us) server_negotiation.cc:407] Connection header received
0810 21:10:03.585923 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.585923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.585932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.585981 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.586407 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.586883 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.587642 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.587818 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.587870 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.589972 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:03.589974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:03.589974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:03.589980 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:03.591373 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.591700 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.591701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.591701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.591750 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.591911 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.591912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.591912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.592005 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:03.592052 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.592167 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.592231 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 21:10:03.592394 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.585616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:37361 (local address 127.0.0.1:35242)
0810 21:10:03.585778 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.585790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.585858 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.586002 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.586004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.586293 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:03.586340 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.586342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.586983 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.586984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.587577 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.587579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.587884 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.587885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.587925 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.588453 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:03.588458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:03.589887 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:03.591440 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:03.591446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:03.591448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:03.591637 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:03.591808 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:03.591808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:03.591809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:03.591862 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:03.592056 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:03.592062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:03.592100 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.592152 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.592278 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":45,"thread_start_us":102,"threads_started":1}
I20260810 21:10:03.594512 8946 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:35242:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 21:10:03.594724 8946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:03.596238 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 76dd87ddb1c647f3bfcc9a378e22fc68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c56c194b45274b98bad6233f1ec03b81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:10:03.596230 9710 tablet_service.cc:1467] Processing CreateTablet for tablet 76dd87ddb1c647f3bfcc9a378e22fc68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c56c194b45274b98bad6233f1ec03b81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:10:03.596230 9455 tablet_service.cc:1467] Processing CreateTablet for tablet 76dd87ddb1c647f3bfcc9a378e22fc68 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c56c194b45274b98bad6233f1ec03b81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 21:10:03.596406 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dd87ddb1c647f3bfcc9a378e22fc68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.596406 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dd87ddb1c647f3bfcc9a378e22fc68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.596410 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dd87ddb1c647f3bfcc9a378e22fc68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.597699 9894 tablet_bootstrap.cc:492] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: Bootstrap starting.
I20260810 21:10:03.597893 9893 tablet_bootstrap.cc:492] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: Bootstrap starting.
I20260810 21:10:03.597910 9885 tablet_bootstrap.cc:492] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: Bootstrap starting.
I20260810 21:10:03.598155 9894 tablet_bootstrap.cc:654] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.598210 9893 tablet_bootstrap.cc:654] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.598325 9885 tablet_bootstrap.cc:654] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.598874 9893 tablet_bootstrap.cc:492] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: No bootstrap required, opened a new log
I20260810 21:10:03.598932 9893 ts_tablet_manager.cc:1397] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.599059 9885 tablet_bootstrap.cc:492] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: No bootstrap required, opened a new log
I20260810 21:10:03.599049 9893 raft_consensus.cc:348] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.599121 9893 raft_consensus.cc:374] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.599136 9893 raft_consensus.cc:729] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.599136 9885 ts_tablet_manager.cc:1397] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.599195 9893 consensus_queue.cc:260] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.599287 9893 ts_tablet_manager.cc:1428] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.599291 9885 raft_consensus.cc:348] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.599373 9885 raft_consensus.cc:374] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.599400 9885 raft_consensus.cc:729] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33157240609e4adcac666cd24499be77, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.599375 9899 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.599440 9899 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.599443 9885 consensus_queue.cc:260] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.599530 9885 ts_tablet_manager.cc:1428] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:03.599606 9897 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.599718 9897 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.599941 9894 tablet_bootstrap.cc:492] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: No bootstrap required, opened a new log
I20260810 21:10:03.599993 9894 ts_tablet_manager.cc:1397] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 21:10:03.600131 9894 raft_consensus.cc:348] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.600214 9894 raft_consensus.cc:374] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.600244 9894 raft_consensus.cc:729] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea8f946143741398434dbd52fe9f3ab, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.600286 9894 consensus_queue.cc:260] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.600391 9894 ts_tablet_manager.cc:1428] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.600458 9900 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.600529 9900 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.603794 9897 raft_consensus.cc:482] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:03.603842 9897 raft_consensus.cc:504] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.603950 9897 leader_election.cc:288] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.604127 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dd87ddb1c647f3bfcc9a378e22fc68" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab" is_pre_election: true
I20260810 21:10:03.604200 9224 raft_consensus.cc:2454] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 0.
I20260810 21:10:03.604173 9748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dd87ddb1c647f3bfcc9a378e22fc68" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7e866667bf4fb4a295b31cb703451e" is_pre_election: true
I20260810 21:10:03.604234 9748 raft_consensus.cc:2454] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 0.
I20260810 21:10:03.604308 9372 leader_election.cc:302] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 5ea8f946143741398434dbd52fe9f3ab; no voters:
I20260810 21:10:03.604390 9897 raft_consensus.cc:2781] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:03.604453 9897 raft_consensus.cc:482] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:03.604482 9897 raft_consensus.cc:3037] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.605114 9897 raft_consensus.cc:504] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.605257 9897 leader_election.cc:288] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.605378 9748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dd87ddb1c647f3bfcc9a378e22fc68" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7e866667bf4fb4a295b31cb703451e"
I20260810 21:10:03.605396 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dd87ddb1c647f3bfcc9a378e22fc68" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab"
I20260810 21:10:03.605446 9224 raft_consensus.cc:3037] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.605448 9748 raft_consensus.cc:3037] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.606108 9748 raft_consensus.cc:2454] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 1.
I20260810 21:10:03.606108 9224 raft_consensus.cc:2454] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 1.
I20260810 21:10:03.606271 9373 leader_election.cc:302] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:03.606377 9897 raft_consensus.cc:2781] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:03.606456 9897 raft_consensus.cc:686] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [term 1 LEADER]: Becoming Leader. State: Replica: 33157240609e4adcac666cd24499be77, State: Running, Role: LEADER
I20260810 21:10:03.606503 9897 consensus_queue.cc:237] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.606781 9944 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: TxnStatusTablet state changed. Reason: New leader 33157240609e4adcac666cd24499be77. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.606863 9944 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:10:03.607061 9969 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 21:10:03.607046 8946 catalog_manager.cc:5495] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33157240609e4adcac666cd24499be77 (127.27.117.66). New cstate: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } health_report { overall_health: UNKNOWN } } }
I20260810 21:10:03.607151 9969 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 21:10:03.613056 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.610653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43515 (local address 127.0.0.1:47588)
0810 21:10:03.610704 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.610715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.610761 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.610921 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.610922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.610923 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:10:03.610965 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.610967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.611583 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.611584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.612128 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.612129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.612495 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.612496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.612581 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.612594 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.612886 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.612896 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.612950 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.612993 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 21:10:03.613049 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.610707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.27.117.66:43515)
0810 21:10:03.610753 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.610755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.610764 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:03.610840 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.610841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.610857 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.610918 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:03.611023 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.611493 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.612213 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.612411 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.612483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.612688 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.612832 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.612903 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.612955 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.612976 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:10:03.614069 9224 raft_consensus.cc:1261] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Refusing update from remote peer 33157240609e4adcac666cd24499be77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.614141 9748 raft_consensus.cc:1261] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Refusing update from remote peer 33157240609e4adcac666cd24499be77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.614286 9944 consensus_queue.cc:1035] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.614379 9897 consensus_queue.cc:1035] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.614993 9900 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: TxnStatusTablet state changed. Reason: New leader 33157240609e4adcac666cd24499be77. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.615091 9900 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.615185 9899 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: TxnStatusTablet state changed. Reason: New leader 33157240609e4adcac666cd24499be77. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.615267 9899 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.616250 9900 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.616336 9900 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 5ea8f946143741398434dbd52fe9f3ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.616315 9897 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.616386 9897 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:10:03.616449 9897 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.616488 9897 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 33157240609e4adcac666cd24499be77: This TxnStatusTablet replica's current role is: LEADER
I20260810 21:10:03.617274 9899 tablet_replica.cc:439] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } } }
I20260810 21:10:03.617367 9899 tablet_replica.cc:442] T 76dd87ddb1c647f3bfcc9a378e22fc68 P 6d7e866667bf4fb4a295b31cb703451e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 21:10:03.620051 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.617883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.27.117.66:43515)
0810 21:10:03.617941 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.617943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.617950 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:10:03.617970 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.617970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.617982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.618034 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:03.618218 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.618656 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.619386 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.619593 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.619670 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.619824 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.619856 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.619912 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.619968 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.619985 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:10:03.624898 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.622444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:38433 (local address 127.27.117.66:43515)
0810 21:10:03.622486 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.622488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.622638 (+ 150us) server_negotiation.cc:407] Connection header received
0810 21:10:03.622698 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.622699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.622707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.622760 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:03.622906 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.623398 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.624134 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.624345 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.624417 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.624589 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.624616 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.624663 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.624729 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.624764 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":22}
I20260810 21:10:03.624958 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.622390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43515 (local address 127.27.117.66:38433)
0810 21:10:03.622575 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:03.622583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:03.622653 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:03.622763 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:03.622764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:03.622765 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:10:03.622826 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.622828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.623475 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.623476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.624049 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:03.624050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.624417 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.624417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:03.624476 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.624493 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.624675 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.624676 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 21:10:03.624727 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:10:03.624807 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260810 21:10:03.626709 9748 raft_consensus.cc:1261] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Refusing update from remote peer 33157240609e4adcac666cd24499be77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.626765 9224 raft_consensus.cc:1261] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Refusing update from remote peer 33157240609e4adcac666cd24499be77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.626919 9978 consensus_queue.cc:1035] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.627048 9944 consensus_queue.cc:1035] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.639106 8945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:03.639194 8945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:03.639781 8945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Sending DeleteTablet for 3 replicas of tablet c2d32dce4468444f8cb71944d2981836
I20260810 21:10:03.640060 9710 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d32dce4468444f8cb71944d2981836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43700
I20260810 21:10:03.640060 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d32dce4468444f8cb71944d2981836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60930
I20260810 21:10:03.640154 9930 tablet_replica.cc:330] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: stopping tablet replica
I20260810 21:10:03.640178 9928 tablet_replica.cc:330] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: stopping tablet replica
I20260810 21:10:03.640216 9930 raft_consensus.cc:2229] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:03.640277 9928 raft_consensus.cc:2229] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:03.640282 9930 raft_consensus.cc:2258] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.640311 9928 raft_consensus.cc:2258] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.640709 9928 ts_tablet_manager.cc:1905] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.642163 9455 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d32dce4468444f8cb71944d2981836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47586
I20260810 21:10:03.642246 9927 tablet_replica.cc:330] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: stopping tablet replica
I20260810 21:10:03.642295 9927 raft_consensus.cc:2229] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:10:03.642380 9927 raft_consensus.cc:2258] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.642426 9928 ts_tablet_manager.cc:1918] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:10:03.642474 9928 log.cc:1196] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/c2d32dce4468444f8cb71944d2981836
I20260810 21:10:03.642706 9928 ts_tablet_manager.cc:1939] T c2d32dce4468444f8cb71944d2981836 P 6d7e866667bf4fb4a295b31cb703451e: Deleting consensus metadata
I20260810 21:10:03.643147 8925 catalog_manager.cc:4841] TS 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767): tablet c2d32dce4468444f8cb71944d2981836 (table test-table [id=b32622020a554cb8a5245f55c92f7bc6]) successfully deleted
I20260810 21:10:03.643254 9930 ts_tablet_manager.cc:1905] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.643340 9927 ts_tablet_manager.cc:1905] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.644536 9927 ts_tablet_manager.cc:1918] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:10:03.644605 9927 log.cc:1196] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/c2d32dce4468444f8cb71944d2981836
I20260810 21:10:03.644871 9930 ts_tablet_manager.cc:1918] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:10:03.644932 9930 log.cc:1196] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/c2d32dce4468444f8cb71944d2981836
I20260810 21:10:03.644938 9927 ts_tablet_manager.cc:1939] T c2d32dce4468444f8cb71944d2981836 P 33157240609e4adcac666cd24499be77: Deleting consensus metadata
I20260810 21:10:03.645159 9930 ts_tablet_manager.cc:1939] T c2d32dce4468444f8cb71944d2981836 P 5ea8f946143741398434dbd52fe9f3ab: Deleting consensus metadata
I20260810 21:10:03.645257 8925 catalog_manager.cc:4841] TS 33157240609e4adcac666cd24499be77 (127.27.117.66:43515): tablet c2d32dce4468444f8cb71944d2981836 (table test-table [id=b32622020a554cb8a5245f55c92f7bc6]) successfully deleted
I20260810 21:10:03.645452 8924 catalog_manager.cc:4841] TS 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565): tablet c2d32dce4468444f8cb71944d2981836 (table test-table [id=b32622020a554cb8a5245f55c92f7bc6]) successfully deleted
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 21:10:03.673091 9031 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:03.666672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.27.117.126:37361)
0810 21:10:03.666747 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.666750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.666893 (+ 143us) server_negotiation.cc:407] Connection header received
0810 21:10:03.666965 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.666966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.666975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.667026 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.667762 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.668364 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.669499 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.669735 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.669790 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.673025 (+ 3235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35258: BlockingRecv error: recv got EOF from 127.0.0.1:35258 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 21:10:03 dist-test-slave-2kcd krb5kdc[8862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM
I20260810 21:10:03.681173 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.673933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.27.117.126:37361)
0810 21:10:03.673979 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.673981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.674217 (+ 236us) server_negotiation.cc:407] Connection header received
0810 21:10:03.674259 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.674260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.674268 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.674313 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:03.674835 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.675254 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.676221 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.676384 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.676432 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.678698 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:03.678700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:03.678700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:03.678708 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:03.680013 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.680360 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.680362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.680362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.680421 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:03.680632 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:03.680633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:03.680633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:03.680739 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:03.680869 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.680983 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.681077 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:10:03.682766 8945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:10:03.682938 8945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:03.684321 9190 tablet_service.cc:1467] Processing CreateTablet for tablet fc0e567240fa4088b07ec2d6113fe260 (DEFAULT_TABLE table=test-table [id=edc782660c804beba21603b6f278821b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.684396 9710 tablet_service.cc:1467] Processing CreateTablet for tablet fc0e567240fa4088b07ec2d6113fe260 (DEFAULT_TABLE table=test-table [id=edc782660c804beba21603b6f278821b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.684396 9455 tablet_service.cc:1467] Processing CreateTablet for tablet fc0e567240fa4088b07ec2d6113fe260 (DEFAULT_TABLE table=test-table [id=edc782660c804beba21603b6f278821b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.684461 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0e567240fa4088b07ec2d6113fe260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.684523 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0e567240fa4088b07ec2d6113fe260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.684523 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0e567240fa4088b07ec2d6113fe260. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.685818 9894 tablet_bootstrap.cc:492] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Bootstrap starting.
I20260810 21:10:03.685886 9893 tablet_bootstrap.cc:492] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Bootstrap starting.
I20260810 21:10:03.686254 9894 tablet_bootstrap.cc:654] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.686288 9893 tablet_bootstrap.cc:654] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.686590 9885 tablet_bootstrap.cc:492] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Bootstrap starting.
I20260810 21:10:03.686928 9893 tablet_bootstrap.cc:492] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: No bootstrap required, opened a new log
I20260810 21:10:03.686982 9885 tablet_bootstrap.cc:654] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.686998 9893 ts_tablet_manager.cc:1397] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.687121 9893 raft_consensus.cc:348] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.687196 9893 raft_consensus.cc:374] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.687229 9893 raft_consensus.cc:729] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.687287 9893 consensus_queue.cc:260] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.687407 9893 ts_tablet_manager.cc:1428] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:03.688108 9885 tablet_bootstrap.cc:492] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: No bootstrap required, opened a new log
I20260810 21:10:03.688162 9885 ts_tablet_manager.cc:1397] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:10:03.688247 9885 raft_consensus.cc:348] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.688308 9885 raft_consensus.cc:374] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.688326 9885 raft_consensus.cc:729] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33157240609e4adcac666cd24499be77, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.688364 9885 consensus_queue.cc:260] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.688452 9885 ts_tablet_manager.cc:1428] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.688793 9894 tablet_bootstrap.cc:492] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: No bootstrap required, opened a new log
I20260810 21:10:03.688846 9894 ts_tablet_manager.cc:1397] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 21:10:03.689006 9894 raft_consensus.cc:348] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.689081 9894 raft_consensus.cc:374] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.689100 9894 raft_consensus.cc:729] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea8f946143741398434dbd52fe9f3ab, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.689138 9894 consensus_queue.cc:260] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.689225 9894 ts_tablet_manager.cc:1428] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.919275 9986 raft_consensus.cc:482] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:03.919425 9986 raft_consensus.cc:504] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.919639 9986 leader_election.cc:288] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.919917 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0e567240fa4088b07ec2d6113fe260" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab" is_pre_election: true
I20260810 21:10:03.919930 9748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0e567240fa4088b07ec2d6113fe260" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7e866667bf4fb4a295b31cb703451e" is_pre_election: true
I20260810 21:10:03.920018 9224 raft_consensus.cc:2454] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 0.
I20260810 21:10:03.920038 9748 raft_consensus.cc:2454] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 0.
I20260810 21:10:03.920192 9372 leader_election.cc:302] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 5ea8f946143741398434dbd52fe9f3ab; no voters:
I20260810 21:10:03.920317 9986 raft_consensus.cc:2781] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:03.920363 9986 raft_consensus.cc:482] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:03.920397 9986 raft_consensus.cc:3037] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.921171 9986 raft_consensus.cc:504] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.921329 9986 leader_election.cc:288] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 election: Requested vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767)
I20260810 21:10:03.921468 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0e567240fa4088b07ec2d6113fe260" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab"
I20260810 21:10:03.921506 9748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0e567240fa4088b07ec2d6113fe260" candidate_uuid: "33157240609e4adcac666cd24499be77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7e866667bf4fb4a295b31cb703451e"
I20260810 21:10:03.921567 9748 raft_consensus.cc:3037] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.921573 9224 raft_consensus.cc:3037] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:03.922199 9748 raft_consensus.cc:2454] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 1.
I20260810 21:10:03.922345 9224 raft_consensus.cc:2454] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33157240609e4adcac666cd24499be77 in term 1.
I20260810 21:10:03.922391 9373 leader_election.cc:302] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:03.922502 9986 raft_consensus.cc:2781] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:03.922585 9986 raft_consensus.cc:686] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 1 LEADER]: Becoming Leader. State: Replica: 33157240609e4adcac666cd24499be77, State: Running, Role: LEADER
I20260810 21:10:03.922657 9986 consensus_queue.cc:237] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.923280 8945 catalog_manager.cc:5495] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33157240609e4adcac666cd24499be77 (127.27.117.66). New cstate: current_term: 1 leader_uuid: "33157240609e4adcac666cd24499be77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } health_report { overall_health: UNKNOWN } } }
I20260810 21:10:03.972909 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:03.970377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.27.117.66:43515)
0810 21:10:03.970425 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:03.970428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:03.970437 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:03.970510 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:03.970511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:03.970522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:03.970587 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:03.970707 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.971308 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.972132 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:03.972395 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:03.972475 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:03.972633 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:03.972668 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:03.972721 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:03.972809 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 21:10:03.972831 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:10:03.973698 9748 raft_consensus.cc:1261] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Refusing update from remote peer 33157240609e4adcac666cd24499be77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.973728 9224 raft_consensus.cc:1261] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Refusing update from remote peer 33157240609e4adcac666cd24499be77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:03.973917 9986 consensus_queue.cc:1035] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.974051 9978 consensus_queue.cc:1035] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:03.975353 9906 mvcc.cc:205] Tried to move back new op lower bound from 7317078851474395136 to 7317078851267481600. Current Snapshot: MvccSnapshot[applied={T|T < 7317078851474395136 or (T in {7317078851474395136})}]
I20260810 21:10:03.976248 8946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:03.976354 8946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:03.976936 8946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Sending DeleteTablet for 3 replicas of tablet fc0e567240fa4088b07ec2d6113fe260
I20260810 21:10:03.977223 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0e567240fa4088b07ec2d6113fe260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60930
I20260810 21:10:03.977247 9455 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0e567240fa4088b07ec2d6113fe260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47586
I20260810 21:10:03.977332 9710 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0e567240fa4088b07ec2d6113fe260 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:03 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43700
I20260810 21:10:03.977334 9930 tablet_replica.cc:330] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: stopping tablet replica
I20260810 21:10:03.977407 9930 raft_consensus.cc:2229] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:03.977444 9928 tablet_replica.cc:330] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: stopping tablet replica
I20260810 21:10:03.977459 9930 raft_consensus.cc:2258] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.977506 9928 raft_consensus.cc:2229] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:03.977536 9927 tablet_replica.cc:330] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: stopping tablet replica
I20260810 21:10:03.977561 9928 raft_consensus.cc:2258] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.977595 9927 raft_consensus.cc:2229] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:10:03.977659 9927 raft_consensus.cc:2258] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:03.977658 8946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:10:03.977793 8946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:03.977937 9930 ts_tablet_manager.cc:1905] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.978006 9928 ts_tablet_manager.cc:1905] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.978328 9927 ts_tablet_manager.cc:1905] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:03.979688 9930 ts_tablet_manager.cc:1918] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:03.979713 9927 ts_tablet_manager.cc:1918] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:03.979748 9930 log.cc:1196] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/fc0e567240fa4088b07ec2d6113fe260
I20260810 21:10:03.979768 9927 log.cc:1196] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/fc0e567240fa4088b07ec2d6113fe260
I20260810 21:10:03.979987 9930 ts_tablet_manager.cc:1939] T fc0e567240fa4088b07ec2d6113fe260 P 5ea8f946143741398434dbd52fe9f3ab: Deleting consensus metadata
I20260810 21:10:03.980003 9927 ts_tablet_manager.cc:1939] T fc0e567240fa4088b07ec2d6113fe260 P 33157240609e4adcac666cd24499be77: Deleting consensus metadata
I20260810 21:10:03.980294 8924 catalog_manager.cc:4841] TS 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565): tablet fc0e567240fa4088b07ec2d6113fe260 (table test-table [id=edc782660c804beba21603b6f278821b]) successfully deleted
I20260810 21:10:03.980305 9928 ts_tablet_manager.cc:1918] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:03.980366 9928 log.cc:1196] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/fc0e567240fa4088b07ec2d6113fe260
I20260810 21:10:03.980590 9928 ts_tablet_manager.cc:1939] T fc0e567240fa4088b07ec2d6113fe260 P 6d7e866667bf4fb4a295b31cb703451e: Deleting consensus metadata
I20260810 21:10:03.980572 9190 tablet_service.cc:1467] Processing CreateTablet for tablet 62925221468041bba9c78aa0e2edd1e3 (DEFAULT_TABLE table=test-table [id=ca0ff78a84364645a9115c73b2105f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.980572 9455 tablet_service.cc:1467] Processing CreateTablet for tablet 62925221468041bba9c78aa0e2edd1e3 (DEFAULT_TABLE table=test-table [id=ca0ff78a84364645a9115c73b2105f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.980674 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62925221468041bba9c78aa0e2edd1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.980674 9190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62925221468041bba9c78aa0e2edd1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.981920 9885 tablet_bootstrap.cc:492] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Bootstrap starting.
I20260810 21:10:03.982019 9710 tablet_service.cc:1467] Processing CreateTablet for tablet 62925221468041bba9c78aa0e2edd1e3 (DEFAULT_TABLE table=test-table [id=ca0ff78a84364645a9115c73b2105f31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:03.982198 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62925221468041bba9c78aa0e2edd1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:03.982224 9885 tablet_bootstrap.cc:654] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.982481 9894 tablet_bootstrap.cc:492] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: Bootstrap starting.
I20260810 21:10:03.982847 9885 tablet_bootstrap.cc:492] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: No bootstrap required, opened a new log
I20260810 21:10:03.982905 9885 ts_tablet_manager.cc:1397] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.983009 9885 raft_consensus.cc:348] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.983080 9885 raft_consensus.cc:374] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.983106 9885 raft_consensus.cc:729] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33157240609e4adcac666cd24499be77, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.983147 9894 tablet_bootstrap.cc:654] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.983157 9885 consensus_queue.cc:260] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.983182 9893 tablet_bootstrap.cc:492] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Bootstrap starting.
I20260810 21:10:03.983263 9885 ts_tablet_manager.cc:1428] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.983363 8925 catalog_manager.cc:4841] TS 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767): tablet fc0e567240fa4088b07ec2d6113fe260 (table test-table [id=edc782660c804beba21603b6f278821b]) successfully deleted
I20260810 21:10:03.983474 8925 catalog_manager.cc:4841] TS 33157240609e4adcac666cd24499be77 (127.27.117.66:43515): tablet fc0e567240fa4088b07ec2d6113fe260 (table test-table [id=edc782660c804beba21603b6f278821b]) successfully deleted
I20260810 21:10:03.983505 9893 tablet_bootstrap.cc:654] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:03.983943 9894 tablet_bootstrap.cc:492] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: No bootstrap required, opened a new log
I20260810 21:10:03.983996 9894 ts_tablet_manager.cc:1397] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:10:03.984149 9894 raft_consensus.cc:348] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.984215 9894 raft_consensus.cc:374] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.984234 9894 raft_consensus.cc:729] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea8f946143741398434dbd52fe9f3ab, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.984233 9893 tablet_bootstrap.cc:492] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: No bootstrap required, opened a new log
I20260810 21:10:03.984280 9893 ts_tablet_manager.cc:1397] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:03.984277 9894 consensus_queue.cc:260] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.984390 9894 ts_tablet_manager.cc:1428] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:03.984381 9893 raft_consensus.cc:348] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.984442 9893 raft_consensus.cc:374] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:03.984465 9893 raft_consensus.cc:729] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Initialized, Role: FOLLOWER
I20260810 21:10:03.984505 9893 consensus_queue.cc:260] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:03.984596 9893 ts_tablet_manager.cc:1428] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:04.029762 9899 raft_consensus.cc:482] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:04.029841 9899 raft_consensus.cc:504] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:04.030010 9899 leader_election.cc:288] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 33157240609e4adcac666cd24499be77 (127.27.117.66:43515)
I20260810 21:10:04.030208 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62925221468041bba9c78aa0e2edd1e3" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33157240609e4adcac666cd24499be77" is_pre_election: true
I20260810 21:10:04.030229 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62925221468041bba9c78aa0e2edd1e3" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab" is_pre_election: true
I20260810 21:10:04.030306 9224 raft_consensus.cc:2454] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 0.
I20260810 21:10:04.030306 9505 raft_consensus.cc:2454] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 0.
I20260810 21:10:04.030481 9642 leader_election.cc:302] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33157240609e4adcac666cd24499be77, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:04.030627 9899 raft_consensus.cc:2781] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:04.030673 9899 raft_consensus.cc:482] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:04.030696 9899 raft_consensus.cc:3037] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:04.031311 9899 raft_consensus.cc:504] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:04.031455 9899 leader_election.cc:288] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 election: Requested vote from peers 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565), 33157240609e4adcac666cd24499be77 (127.27.117.66:43515)
I20260810 21:10:04.031625 9224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62925221468041bba9c78aa0e2edd1e3" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ea8f946143741398434dbd52fe9f3ab"
I20260810 21:10:04.031631 9505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62925221468041bba9c78aa0e2edd1e3" candidate_uuid: "6d7e866667bf4fb4a295b31cb703451e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33157240609e4adcac666cd24499be77"
I20260810 21:10:04.031697 9224 raft_consensus.cc:3037] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:04.031718 9505 raft_consensus.cc:3037] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:04.032261 9224 raft_consensus.cc:2454] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 1.
I20260810 21:10:04.032281 9505 raft_consensus.cc:2454] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7e866667bf4fb4a295b31cb703451e in term 1.
I20260810 21:10:04.032433 9641 leader_election.cc:302] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ea8f946143741398434dbd52fe9f3ab, 6d7e866667bf4fb4a295b31cb703451e; no voters:
I20260810 21:10:04.032572 9899 raft_consensus.cc:2781] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:04.032687 9899 raft_consensus.cc:686] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 1 LEADER]: Becoming Leader. State: Replica: 6d7e866667bf4fb4a295b31cb703451e, State: Running, Role: LEADER
I20260810 21:10:04.032745 9899 consensus_queue.cc:237] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } }
I20260810 21:10:04.033346 8946 catalog_manager.cc:5495] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67). New cstate: current_term: 1 leader_uuid: "6d7e866667bf4fb4a295b31cb703451e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d7e866667bf4fb4a295b31cb703451e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 36767 } health_report { overall_health: HEALTHY } } }
I20260810 21:10:04.045708 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:04.043459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.27.117.67:36767)
0810 21:10:04.043502 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:04.043505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:04.043514 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:04.043581 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:04.043582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:04.043593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:04.043641 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:04.043757 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.044236 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.045032 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.045227 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.045301 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:04.045450 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:04.045489 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:04.045544 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:04.045608 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 21:10:04.045631 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:10:04.050128 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:04.048063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45493 (local address 127.27.117.66:43515)
0810 21:10:04.048132 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:04.048134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:04.048183 (+ 49us) server_negotiation.cc:407] Connection header received
0810 21:10:04.048239 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:04.048240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:04.048247 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:04.048284 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:04.048384 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.048886 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.049567 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.049749 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.049797 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:04.049952 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:04.049975 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:04.050011 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:04.050070 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 21:10:04.050084 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:10:04.050153 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:04.047974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43515 (local address 127.27.117.67:45493)
0810 21:10:04.048134 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:04.048143 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:04.048191 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:04.048298 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:04.048299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:04.048300 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:10:04.048334 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:04.048335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.048946 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.048946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:04.049493 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:04.049494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.049811 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.049812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:04.049848 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:04.049883 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:10:04.050035 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:10:04.050037 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:04.050069 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 21:10:04.050103 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260810 21:10:04.051910 9224 raft_consensus.cc:1261] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Refusing update from remote peer 6d7e866667bf4fb4a295b31cb703451e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:04.051929 9505 raft_consensus.cc:1261] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Refusing update from remote peer 6d7e866667bf4fb4a295b31cb703451e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:04.052143 9899 consensus_queue.cc:1035] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ea8f946143741398434dbd52fe9f3ab" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 46565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:04.052277 10035 consensus_queue.cc:1035] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [LEADER]: Connected to new peer: Peer: permanent_uuid: "33157240609e4adcac666cd24499be77" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:04.058841 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:04.056824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:37215 (local address 127.27.117.67:36767)
0810 21:10:04.056874 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:04.056876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:04.056884 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:04.056905 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:04.056906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:04.056917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:04.056972 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:04.057104 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.057540 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.058200 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.058365 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.058450 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:04.058599 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:04.058631 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:04.058686 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:04.058752 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 21:10:04.058774 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:10:04.058877 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:04.056714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:36767 (local address 127.27.117.66:37215)
0810 21:10:04.056753 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:04.056763 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:04.056818 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:04.056992 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:04.056993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:04.056994 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 21:10:04.057038 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:04.057040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.057601 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.057602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:04.058128 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:04.058130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:04.058448 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:04.058449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:04.058523 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:04.058531 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 21:10:04.058699 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 21:10:04.058702 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:10:04.058750 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 21:10:04.058802 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 21:10:04.064723 8946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:04.064855 8946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:04.065485 8946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f96dceb453e4415a9a911e44120fe299: Sending DeleteTablet for 3 replicas of tablet 62925221468041bba9c78aa0e2edd1e3
I20260810 21:10:04.065755 9190 tablet_service.cc:1514] Processing DeleteTablet for tablet 62925221468041bba9c78aa0e2edd1e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:04 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60930
I20260810 21:10:04.065809 9710 tablet_service.cc:1514] Processing DeleteTablet for tablet 62925221468041bba9c78aa0e2edd1e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:04 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43700
I20260810 21:10:04.065896 9928 tablet_replica.cc:330] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: stopping tablet replica
I20260810 21:10:04.065894 9455 tablet_service.cc:1514] Processing DeleteTablet for tablet 62925221468041bba9c78aa0e2edd1e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:04 UTC) from {username='oryx', principal='oryx/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47586
I20260810 21:10:04.065897 9930 tablet_replica.cc:330] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: stopping tablet replica
I20260810 21:10:04.065958 9930 raft_consensus.cc:2229] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:04.065956 9928 raft_consensus.cc:2229] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:10:04.065964 9927 tablet_replica.cc:330] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: stopping tablet replica
I20260810 21:10:04.066009 9930 raft_consensus.cc:2258] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:04.066013 9927 raft_consensus.cc:2229] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:04.066026 9928 raft_consensus.cc:2258] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:04.066056 9927 raft_consensus.cc:2258] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:04.066339 9930 ts_tablet_manager.cc:1905] T 62925221468041bba9c78aa0e2edd1e3 P 5ea8f946143741398434dbd52fe9f3ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:04.066341 9928 ts_tablet_manager.cc:1905] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:04.066391 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 9078
I20260810 21:10:04.077435 9928 ts_tablet_manager.cc:1918] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:10:04.077503 9928 log.cc:1196] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/62925221468041bba9c78aa0e2edd1e3
I20260810 21:10:04.077697 9928 ts_tablet_manager.cc:1939] T 62925221468041bba9c78aa0e2edd1e3 P 6d7e866667bf4fb4a295b31cb703451e: Deleting consensus metadata
I20260810 21:10:04.078087 8925 catalog_manager.cc:4841] TS 6d7e866667bf4fb4a295b31cb703451e (127.27.117.67:36767): tablet 62925221468041bba9c78aa0e2edd1e3 (table test-table [id=ca0ff78a84364645a9115c73b2105f31]) successfully deleted
I20260810 21:10:04.079133 9927 ts_tablet_manager.cc:1905] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:04.080844 9927 ts_tablet_manager.cc:1918] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 21:10:04.080922 9927 log.cc:1196] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/62925221468041bba9c78aa0e2edd1e3
I20260810 21:10:04.081168 9927 ts_tablet_manager.cc:1939] T 62925221468041bba9c78aa0e2edd1e3 P 33157240609e4adcac666cd24499be77: Deleting consensus metadata
I20260810 21:10:04.081568 8925 catalog_manager.cc:4841] TS 33157240609e4adcac666cd24499be77 (127.27.117.66:43515): tablet 62925221468041bba9c78aa0e2edd1e3 (table test-table [id=ca0ff78a84364645a9115c73b2105f31]) successfully deleted
W20260810 21:10:04.086354 8924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:46565 (error 108)
I20260810 21:10:04.086670 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 9331
I20260810 21:10:04.092218 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:04.086763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:46565 (local address 127.0.0.1:60940)
0810 21:10:04.092087 (+ 5324us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:04.092159 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:46565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5300,"thread_start_us":41,"threads_started":1}
W20260810 21:10:04.092371 8924 catalog_manager.cc:4584] TS 5ea8f946143741398434dbd52fe9f3ab (127.27.117.65:46565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62925221468041bba9c78aa0e2edd1e3: Network error: Client connection negotiation failed: client connection to 127.27.117.65:46565: connect: Connection refused (error 111)
I20260810 21:10:04.103497 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 9607
I20260810 21:10:04.113860 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 8890
2026-08-10T21:10:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3056 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:10:04 dist-test-slave-2kcd krb5kdc[10058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:10:04 dist-test-slave-2kcd krb5kdc[10058](info): set up 2 sockets
Aug 10 21:10:04 dist-test-slave-2kcd krb5kdc[10058](info): commencing operation
krb5kdc: starting...
W20260810 21:10:06.381700 28117 mini_kdc.cc:121] Time spent starting KDC: real 2.239s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:10:06 dist-test-slave-2kcd krb5kdc[10058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:10:06.625036 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42731
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:06.706303 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:06.706532 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:06.706573 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:06.707700 10086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:06.707760 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:06.707783 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:06.707803 10086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:06.707834 10086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:06.709354 10086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:06.709630 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:06.709852 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:06.714685 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:06.714691 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:06.714695 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:06.714891 10086 server_base.cc:1034] running on GCE node
I20260810 21:10:06.715190 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:06.715478 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:06.716645 10086 hybrid_clock.cc:648] HybridClock initialized: now 1786396206716637 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T21:10:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2600 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:10:06 dist-test-slave-2kcd krb5kdc[10108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:10:06 dist-test-slave-2kcd krb5kdc[10108](info): set up 2 sockets
Aug 10 21:10:06 dist-test-slave-2kcd krb5kdc[10108](info): commencing operation
krb5kdc: starting...
W20260810 21:10:08.004360 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:10:08.123276 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35057
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:08.202412 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:08.202632 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:08.202672 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:08.203769 10133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:08.203836 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:08.203877 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:08.203922 10133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:08.203949 10133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:08.205461 10133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:08.205731 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:08.205953 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:08.210613 10140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.210650 10139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.210678 10133 server_base.cc:1034] running on GCE node
W20260810 21:10:08.210637 10144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.211048 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:08.211328 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:08.212469 10133 hybrid_clock.cc:648] HybridClock initialized: now 1786396208212448 us; error 37 us; skew 500 ppm
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:08.231864 10133 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:10:08.232283 10133 webserver.cc:460] Webserver started at http://127.27.117.126:37911/ using document root <none> and password file <none>
I20260810 21:10:08.232522 10133 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:08.232605 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:08.232874 10133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:08.233886 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "971dc7a18fbd477abdc1eb6349e3db78"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa92715e376a498c67a299275f009ef"
server_key_iv: "4d20220f641a355f6ab44248aa4107cb"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.234249 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "971dc7a18fbd477abdc1eb6349e3db78"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa92715e376a498c67a299275f009ef"
server_key_iv: "4d20220f641a355f6ab44248aa4107cb"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.235561 10133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:10:08.236316 10152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.236495 10133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.236603 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "971dc7a18fbd477abdc1eb6349e3db78"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa92715e376a498c67a299275f009ef"
server_key_iv: "4d20220f641a355f6ab44248aa4107cb"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.236686 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:08.267148 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:08.268904 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:08.269080 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:08.273164 10133 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35057
I20260810 21:10:08.273212 10249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35057 every 8 connection(s)
I20260810 21:10:08.273548 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:10:08.274358 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:08.276466 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Bootstrap starting.
I20260810 21:10:08.277081 10251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:08.277392 10251 log.cc:824] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:08.278040 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: No bootstrap required, opened a new log
I20260810 21:10:08.278208 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 10133
I20260810 21:10:08.278345 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:10:08.278534 28117 external_mini_cluster.cc:1428] Setting key 20830d3fc95c8eb2ec5003b85fda23c5
I20260810 21:10:08.280028 10251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:08.280187 10251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:08.280225 10251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971dc7a18fbd477abdc1eb6349e3db78, State: Initialized, Role: FOLLOWER
I20260810 21:10:08.280359 10251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:08.280444 10251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:10:08.280479 10251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:10:08.280529 10251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:08.281530 10251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:08.281617 10251 leader_election.cc:302] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971dc7a18fbd477abdc1eb6349e3db78; no voters:
I20260810 21:10:08.281790 10251 leader_election.cc:288] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:10:08.281845 10258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:08.281984 10258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 LEADER]: Becoming Leader. State: Replica: 971dc7a18fbd477abdc1eb6349e3db78, State: Running, Role: LEADER
I20260810 21:10:08.282049 10251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:10:08.282073 10258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:08.282379 10258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971dc7a18fbd477abdc1eb6349e3db78. Latest consensus state: current_term: 1 leader_uuid: "971dc7a18fbd477abdc1eb6349e3db78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } } }
I20260810 21:10:08.282467 10258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:08.282385 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971dc7a18fbd477abdc1eb6349e3db78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } } }
I20260810 21:10:08.282629 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: This master's current role is: LEADER
W20260810 21:10:08.283824 10289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:10:08.283926 10289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:10:08.284055 10274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:08.284341 10274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:08.286332 10274 catalog_manager.cc:1347] Generated new cluster ID: 57f040a5ec3b4b519dc0d2b6b88a9e9a
I20260810 21:10:08.286377 10274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:10:08.290352 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.279025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.27.117.126:35057)
0810 21:10:08.279228 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:08.279235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:08.279258 (+ 23us) server_negotiation.cc:407] Connection header received
0810 21:10:08.280014 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:08.280020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:08.280065 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:08.280171 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:08.281026 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.282158 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.283201 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.283859 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.284246 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.286746 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:08.286749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:08.286750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:08.286759 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:08.288519 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.288926 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.288928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.288929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.289041 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.289214 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.289215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.289215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.289475 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:08.289615 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:08.290034 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 21:10:08.290155 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260810 21:10:08.306416 10274 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:10:08.307101 10274 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:08.314961 10274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Generated new TSK 0
I20260810 21:10:08.315160 10274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:10:08.346984 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35057
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:08.428514 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:08.428730 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:08.428752 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:08.429925 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:08.429967 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:08.430018 10299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:10:08.431500 10299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:08.431727 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:08.431953 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:08.436879 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.436887 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.437150 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.437299 10299 server_base.cc:1034] running on GCE node
I20260810 21:10:08.437503 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:08.437750 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:08.438925 10299 hybrid_clock.cc:648] HybridClock initialized: now 1786396208438886 us; error 47 us; skew 500 ppm
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:08.441350 10299 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:10:08.441766 10299 webserver.cc:460] Webserver started at http://127.27.117.65:34439/ using document root <none> and password file <none>
I20260810 21:10:08.441988 10299 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:08.442056 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:08.442220 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:08.443208 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "7e86a39b649f459c88c010b36956dd50"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa9e6f65fdab2dcf933dc6c1493a826"
server_key_iv: "5d87eed6cf226a04e81bb5b47b28fe18"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.443574 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "7e86a39b649f459c88c010b36956dd50"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa9e6f65fdab2dcf933dc6c1493a826"
server_key_iv: "5d87eed6cf226a04e81bb5b47b28fe18"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.444902 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:10:08.445632 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.445796 10299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:08.445894 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "7e86a39b649f459c88c010b36956dd50"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa9e6f65fdab2dcf933dc6c1493a826"
server_key_iv: "5d87eed6cf226a04e81bb5b47b28fe18"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.445986 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:08.458319 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:08.460011 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:08.460175 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:08.460402 10299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:08.460695 10299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:08.460748 10299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.460824 10299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:08.460873 10299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.467000 10299 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:45377
I20260810 21:10:08.467096 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:45377 every 8 connection(s)
I20260810 21:10:08.467410 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:10:08.471698 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 10299
I20260810 21:10:08.471819 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:10:08.472056 28117 external_mini_cluster.cc:1428] Setting key 2083ccdc75f098f6d319f6463eb9820c
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:08.477375 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.468131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:53797 (local address 127.27.117.126:35057)
0810 21:10:08.468192 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:08.468194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:08.468414 (+ 220us) server_negotiation.cc:407] Connection header received
0810 21:10:08.469028 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:08.469029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:08.469047 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:08.469092 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:08.470051 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.470571 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.471599 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.471851 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.471925 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.474438 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:08.474440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:08.474441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:08.474451 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:08.475902 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.476325 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.476326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.476327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.476389 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.476656 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.476656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.476657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.476769 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:08.476876 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:08.477183 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 21:10:08.477283 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:10:08.477587 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.468107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:53797)
0810 21:10:08.468266 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:08.468321 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:08.468912 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:08.469157 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:08.469161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:08.469745 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:08.469932 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:08.469937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.470674 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.470676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:08.471468 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:08.471470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.471942 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.471944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:08.472105 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.472696 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:08.472705 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:08.474337 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:08.476005 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:08.476013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:08.476015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:08.476246 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:08.476459 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:08.476460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:08.476460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:08.476578 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:08.476867 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:08.476875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:08.477028 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 21:10:08.477143 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 21:10:08.477306 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260810 21:10:08.478108 10537 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35057
I20260810 21:10:08.478191 10537 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:08.478384 10537 heartbeater.cc:511] Master 127.27.117.126:35057 requested a full tablet report, sending...
I20260810 21:10:08.478961 10184 ts_manager.cc:194] Registered new tserver with Master: 7e86a39b649f459c88c010b36956dd50 (127.27.117.65:45377)
I20260810 21:10:08.479589 10184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:53797
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:10:08.497161 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35057
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:08.578920 10550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:08.579125 10550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:08.579150 10550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:08.580319 10550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:08.580363 10550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:08.580389 10550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:10:08.581988 10550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:08.582253 10550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:08.582517 10550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:08.587635 10563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.587648 10561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.587644 10560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.587977 10550 server_base.cc:1034] running on GCE node
I20260810 21:10:08.588126 10550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:08.588304 10550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:08.589433 10550 hybrid_clock.cc:648] HybridClock initialized: now 1786396208589420 us; error 28 us; skew 500 ppm
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:08.591744 10550 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:10:08.592160 10550 webserver.cc:460] Webserver started at http://127.27.117.66:34189/ using document root <none> and password file <none>
I20260810 21:10:08.592373 10550 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:08.592417 10550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:08.592613 10550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:08.593690 10550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "8ff86e11cd044a77895b22736f947457"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "e838a5dd0bd153a7746c82f03e96a9a2"
server_key_iv: "dd389987a8766aace68654eb6cc09f44"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.594075 10550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "8ff86e11cd044a77895b22736f947457"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "e838a5dd0bd153a7746c82f03e96a9a2"
server_key_iv: "dd389987a8766aace68654eb6cc09f44"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.595484 10550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:10:08.596261 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.596449 10550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:08.596557 10550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "8ff86e11cd044a77895b22736f947457"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "e838a5dd0bd153a7746c82f03e96a9a2"
server_key_iv: "dd389987a8766aace68654eb6cc09f44"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.596629 10550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:08.613698 10550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:08.615530 10550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:08.615693 10550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:08.615931 10550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:08.616272 10550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:08.616326 10550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.616359 10550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:08.616387 10550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.622189 10550 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:44263
I20260810 21:10:08.622249 10775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:44263 every 8 connection(s)
I20260810 21:10:08.622605 10550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:08.631594 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.623362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:44255 (local address 127.27.117.126:35057)
0810 21:10:08.623434 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:08.623437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:08.623662 (+ 225us) server_negotiation.cc:407] Connection header received
0810 21:10:08.624200 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:08.624201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:08.624213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:08.624260 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:08.625281 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.625775 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.626806 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.627028 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.627116 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.629151 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:08.629152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:08.629153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:08.629159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:08.630540 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.630874 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.630875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.630875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.630929 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.631134 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.631135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.631135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.631231 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:08.631282 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:08.631473 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 21:10:08.631533 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 21:10:08.631816 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.623336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:44255)
0810 21:10:08.623542 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:08.623578 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:08.624085 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:08.624338 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:08.624342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:08.624983 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:08.625175 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:08.625179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.625872 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.625873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:08.626696 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:08.626699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.627127 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.627128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:08.627249 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.627648 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:08.627655 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:08.629053 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:08.630610 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:08.630615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:08.630617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:08.630810 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:08.630991 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:08.630991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:08.630991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:08.631081 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:08.631284 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:08.631289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:08.631388 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 21:10:08.631464 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 21:10:08.631572 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260810 21:10:08.632197 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 10550
I20260810 21:10:08.632306 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:10:08.632314 10777 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35057
I20260810 21:10:08.632411 10777 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:08.632543 28117 external_mini_cluster.cc:1428] Setting key c2128ff721fb798d5e46a8da14bc8388
I20260810 21:10:08.632598 10777 heartbeater.cc:511] Master 127.27.117.126:35057 requested a full tablet report, sending...
I20260810 21:10:08.632988 10183 ts_manager.cc:194] Registered new tserver with Master: 8ff86e11cd044a77895b22736f947457 (127.27.117.66:44263)
I20260810 21:10:08.633490 10183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:44255
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:10:08.657759 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35057
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:08.736739 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:08.736932 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:08.736954 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:08.738121 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:08.738164 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:08.738219 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:10:08.739753 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:08.739976 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:08.740219 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:08.744952 10797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.744983 10798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.745075 10785 server_base.cc:1034] running on GCE node
W20260810 21:10:08.744983 10800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.745349 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:08.745595 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:08.746734 10785 hybrid_clock.cc:648] HybridClock initialized: now 1786396208746723 us; error 30 us; skew 500 ppm
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:08.749296 10785 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:10:08.749689 10785 webserver.cc:460] Webserver started at http://127.27.117.67:39527/ using document root <none> and password file <none>
I20260810 21:10:08.749914 10785 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:08.749987 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:08.750135 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:08.751089 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "94fb67310b344653b0304a16cd142245"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0c0a17a283ad6ee1cf51f906f8f88f96"
server_key_iv: "1e472bd85b345c8f19ec06a320835db4"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.751435 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "94fb67310b344653b0304a16cd142245"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0c0a17a283ad6ee1cf51f906f8f88f96"
server_key_iv: "1e472bd85b345c8f19ec06a320835db4"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.752769 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:10:08.753499 10813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.753679 10785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:10:08.753775 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "94fb67310b344653b0304a16cd142245"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0c0a17a283ad6ee1cf51f906f8f88f96"
server_key_iv: "1e472bd85b345c8f19ec06a320835db4"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.753854 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:08.762517 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:08.764113 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:08.764261 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:08.764456 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:08.764724 10785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:08.764775 10785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.764828 10785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:08.764858 10785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.770416 10785 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:33579
I20260810 21:10:08.770486 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:33579 every 8 connection(s)
I20260810 21:10:08.770757 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:10:08.772289 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 10785
I20260810 21:10:08.772372 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:10:08.772553 28117 external_mini_cluster.cc:1428] Setting key 26203d88a98744cbe57bd32cd2d2a5bc
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:08.780107 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.771415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:55779 (local address 127.27.117.126:35057)
0810 21:10:08.771486 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:08.771489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:08.771660 (+ 171us) server_negotiation.cc:407] Connection header received
0810 21:10:08.772067 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:08.772067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:08.772077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:08.772165 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:08.773321 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.773927 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.774884 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.775095 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.775170 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.777177 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:08.777179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:08.777179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:08.777186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:08.778600 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.779129 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.779130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.779130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.779207 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:08.779506 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:08.779507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:08.779508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:08.779641 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:08.779707 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:08.779979 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 21:10:08.780047 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:10:08.780246 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:08.771387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:55779)
0810 21:10:08.771532 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:08.771578 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:08.771985 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:08.772207 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:08.772210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:08.772953 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:08.773223 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:08.773228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.774025 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.774026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:08.774795 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:08.774797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:08.775179 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:08.775180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:08.775304 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:08.775774 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:08.775782 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:08.777073 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:08.778703 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:08.778710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:08.778712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:08.779044 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:08.779269 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:08.779271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:08.779271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:08.779412 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:08.779717 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:08.779723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:08.779862 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 21:10:08.779948 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 21:10:08.780060 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260810 21:10:08.780743 11020 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35057
I20260810 21:10:08.780853 11020 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:08.781033 11020 heartbeater.cc:511] Master 127.27.117.126:35057 requested a full tablet report, sending...
I20260810 21:10:08.781395 10183 ts_manager.cc:194] Registered new tserver with Master: 94fb67310b344653b0304a16cd142245 (127.27.117.67:33579)
I20260810 21:10:08.781831 10183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:55779
I20260810 21:10:08.784346 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab.
Entry for principal oryx/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab.
WARNING: no policy specified for oryx/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab.
Entry for principal oryx/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab.
WARNING: no policy specified for oryx/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab.
Entry for principal oryx/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab.
I20260810 21:10:08.851926 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 10299
I20260810 21:10:08.869704 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 10550
I20260810 21:10:08.884567 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 10785
I20260810 21:10:08.890404 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 10133
I20260810 21:10:08.902417 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35057
--webserver_interface=127.27.117.126
--webserver_port=37911
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:08.982796 11037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:08.983045 11037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:08.983086 11037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:08.984217 11037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:08.984288 11037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:08.984313 11037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:08.984342 11037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:08.984381 11037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:08.985855 11037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=37911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:08.986128 11037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:08.986349 11037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:08.991179 11043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.991242 11037 server_base.cc:1034] running on GCE node
W20260810 21:10:08.991158 11048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:08.991179 11044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:08.991633 11037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:08.991936 11037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:08.993080 11037 hybrid_clock.cc:648] HybridClock initialized: now 1786396208993065 us; error 42 us; skew 500 ppm
Aug 10 21:10:08 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396208, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:08.995651 11037 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:10:08.996102 11037 webserver.cc:460] Webserver started at http://127.27.117.126:37911/ using document root <none> and password file <none>
I20260810 21:10:08.996400 11037 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:08.996460 11037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:08.997807 11037 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:08.998486 11058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:08.998728 11037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:08.998795 11037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "971dc7a18fbd477abdc1eb6349e3db78"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa92715e376a498c67a299275f009ef"
server_key_iv: "4d20220f641a355f6ab44248aa4107cb"
server_key_version: "encryptionkey@0"
I20260810 21:10:08.999152 11037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:09.017854 11037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:09.020195 11037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:09.020356 11037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:09.024520 11037 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35057
I20260810 21:10:09.024554 11142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35057 every 8 connection(s)
I20260810 21:10:09.024976 11037 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:10:09.026409 11143 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:10:09.027074 11143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Bootstrap starting.
I20260810 21:10:09.027523 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 11037
I20260810 21:10:09.027841 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:45377
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:35057
--webserver_port=34439
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:10:09.036964 11143 log.cc:824] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:09.038714 11143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:10:09.039054 11143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Bootstrap complete.
I20260810 21:10:09.041469 11143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:09.041688 11143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971dc7a18fbd477abdc1eb6349e3db78, State: Initialized, Role: FOLLOWER
I20260810 21:10:09.041877 11143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:09.041940 11143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:10:09.042006 11143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:10:09.042053 11143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 1 FOLLOWER]: Advancing to term 2
I20260810 21:10:09.043108 11143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:09.043196 11143 leader_election.cc:302] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971dc7a18fbd477abdc1eb6349e3db78; no voters:
I20260810 21:10:09.043393 11143 leader_election.cc:288] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 21:10:09.043428 11148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 21:10:09.043558 11148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [term 2 LEADER]: Becoming Leader. State: Replica: 971dc7a18fbd477abdc1eb6349e3db78, State: Running, Role: LEADER
I20260810 21:10:09.043668 11143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:10:09.043676 11148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } }
I20260810 21:10:09.044056 11158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "971dc7a18fbd477abdc1eb6349e3db78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } } }
I20260810 21:10:09.044070 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971dc7a18fbd477abdc1eb6349e3db78. Latest consensus state: current_term: 2 leader_uuid: "971dc7a18fbd477abdc1eb6349e3db78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971dc7a18fbd477abdc1eb6349e3db78" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35057 } } }
I20260810 21:10:09.044219 11158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:09.044242 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:09.044638 11170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:09.045205 11170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:10:09.045334 11170 catalog_manager.cc:1259] Loaded cluster ID: 57f040a5ec3b4b519dc0d2b6b88a9e9a
I20260810 21:10:09.045372 11170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:10:09.046564 11170 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:09.046738 11170 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 971dc7a18fbd477abdc1eb6349e3db78: Loaded TSK: 0
I20260810 21:10:09.046962 11170 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 21:10:09.112301 11145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:09.112489 11145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:09.112510 11145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:09.113763 11145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:09.113804 11145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:09.113829 11145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:10:09.115223 11145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.65.keytab
--principal=oryx/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:45377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=34439
--tserver_master_addrs=127.27.117.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:09.115445 11145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:09.115702 11145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:09.121039 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:09.121172 11199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:09.121263 11145 server_base.cc:1034] running on GCE node
W20260810 21:10:09.121048 11192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:09.121603 11145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:09.121886 11145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:09.123086 11145 hybrid_clock.cc:648] HybridClock initialized: now 1786396209123067 us; error 33 us; skew 500 ppm
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396209, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:09.125993 11145 init.cc:377] Logged in from keytab as oryx/127.27.117.65@KRBTEST.COM (short username oryx)
I20260810 21:10:09.126497 11145 webserver.cc:460] Webserver started at http://127.27.117.65:34439/ using document root <none> and password file <none>
I20260810 21:10:09.126729 11145 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:09.126809 11145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:09.128207 11145 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:09.128885 11208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.129055 11145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:09.129165 11145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "7e86a39b649f459c88c010b36956dd50"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0aa9e6f65fdab2dcf933dc6c1493a826"
server_key_iv: "5d87eed6cf226a04e81bb5b47b28fe18"
server_key_version: "encryptionkey@0"
I20260810 21:10:09.129490 11145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:09.167645 11145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:09.170008 11145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:09.170132 11145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:09.170389 11145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:09.170717 11145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:09.170751 11145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.170773 11145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:09.170819 11145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.176843 11145 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:45377
I20260810 21:10:09.176892 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:45377 every 8 connection(s)
I20260810 21:10:09.177207 11145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.184062 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 11145
I20260810 21:10:09.184358 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:44263
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:35057
--webserver_port=34189
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:10:09.184886 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.177829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:44271)
0810 21:10:09.177986 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.178024 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.178479 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.178833 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.178836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.179393 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.179578 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.179583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.180633 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.180634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.181380 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.181382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.181899 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.181899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.181989 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.182406 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.182413 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.184662 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":100,"threads_started":1}
W20260810 21:10:09.184952 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.177915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:44271 (local address 127.27.117.126:35057)
0810 21:10:09.178093 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.178101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.178135 (+ 34us) server_negotiation.cc:407] Connection header received
0810 21:10:09.178632 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.178637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.178675 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.178781 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.179679 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.180549 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.181455 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.181821 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.181898 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.184776 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:44271: BlockingRecv error: recv got EOF from 127.27.117.65:44271 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
W20260810 21:10:09.185092 11427 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:35057 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.117.126:35057: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.190034 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.185322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:49067)
0810 21:10:09.185367 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.185377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.185411 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.185546 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.185546 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.185748 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.185799 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.185801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.186631 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.186632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.187101 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.187102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.187594 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.187595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.187658 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.188055 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.188059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.189922 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 21:10:09.190058 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.185370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:49067 (local address 127.27.117.126:35057)
0810 21:10:09.185404 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.185406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.185415 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:09.185473 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.185474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.185482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.185527 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.185863 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.186546 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.187180 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.187508 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.187579 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.189999 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:49067: BlockingRecv error: recv got EOF from 127.27.117.65:49067 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.194689 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.190352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:43571)
0810 21:10:09.190399 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.190413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.190460 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.190574 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.190575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.190936 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.190965 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.190968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.191616 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.191617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.192088 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.192089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.192464 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.192466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.192519 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.192944 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.192949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.194607 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 21:10:09.194712 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.190419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:43571 (local address 127.27.117.126:35057)
0810 21:10:09.190480 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.190482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.190490 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:09.190509 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.190510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.190519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.190576 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.191082 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.191531 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.192172 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.192374 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.192460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.194678 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:43571: BlockingRecv error: recv got EOF from 127.27.117.65:43571 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 21:10:09.194820 11427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:10:09.267405 11432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:09.267583 11432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:09.267606 11432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:09.268748 11432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:09.268843 11432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:09.268893 11432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:10:09.270356 11432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.66.keytab
--principal=oryx/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:44263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=34189
--tserver_master_addrs=127.27.117.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:09.270593 11432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:09.270831 11432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:09.275635 11445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:09.275661 11439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:09.275640 11440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:09.275681 11432 server_base.cc:1034] running on GCE node
I20260810 21:10:09.276119 11432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:09.276343 11432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:09.277500 11432 hybrid_clock.cc:648] HybridClock initialized: now 1786396209277487 us; error 41 us; skew 500 ppm
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396209, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:09.279912 11432 init.cc:377] Logged in from keytab as oryx/127.27.117.66@KRBTEST.COM (short username oryx)
I20260810 21:10:09.280340 11432 webserver.cc:460] Webserver started at http://127.27.117.66:34189/ using document root <none> and password file <none>
I20260810 21:10:09.280581 11432 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:09.280668 11432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:09.282073 11432 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:09.282675 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.282836 11432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:09.282936 11432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "8ff86e11cd044a77895b22736f947457"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "e838a5dd0bd153a7746c82f03e96a9a2"
server_key_iv: "dd389987a8766aace68654eb6cc09f44"
server_key_version: "encryptionkey@0"
I20260810 21:10:09.283304 11432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:09.296660 11432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:09.298527 11432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:09.298693 11432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:09.298920 11432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:09.299273 11432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:09.299329 11432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.299371 11432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:09.299399 11432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.305675 11432 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:44263
I20260810 21:10:09.305732 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:44263 every 8 connection(s)
I20260810 21:10:09.306064 11432 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:10:09.310235 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 11432
I20260810 21:10:09.310523 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:33579
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:35057
--webserver_port=39527
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 21:10:09.313815 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.306765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:53749 (local address 127.27.117.126:35057)
0810 21:10:09.306849 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.306851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.307060 (+ 209us) server_negotiation.cc:407] Connection header received
0810 21:10:09.307547 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.307548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.307561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.307618 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.308493 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.309097 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.310123 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.310338 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.310402 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.313758 (+ 3356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:53749: BlockingRecv error: recv got EOF from 127.27.117.66:53749 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 21:10:09.313868 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.306734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:53749)
0810 21:10:09.306929 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.306972 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.307461 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.307677 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.307680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.308234 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.308416 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.308420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.309177 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.309178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.310001 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.310004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.310416 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.310417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.310551 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.311114 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.311122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.313669 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":123,"threads_started":1}
W20260810 21:10:09.314055 11661 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:35057 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.117.126:35057: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.318838 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.314310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:33675)
0810 21:10:09.314353 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.314362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.314398 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.314547 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.314548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.314656 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.314697 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.314698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.315641 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.315641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.316126 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.316128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.316618 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.316619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.316663 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.317139 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.317144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.318792 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 21:10:09.318995 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.314340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:33675 (local address 127.27.117.126:35057)
0810 21:10:09.314418 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.314421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.314432 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:10:09.314458 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.314459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.314470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.314537 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.314772 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.315560 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.316210 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.316541 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.316623 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.318916 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:33675: BlockingRecv error: recv got EOF from 127.27.117.66:33675 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.323094 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.319118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:38989)
0810 21:10:09.319207 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.319217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.319255 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.319383 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.319384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.319499 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.319523 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.319524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.320129 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.320130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.320599 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.320600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.321019 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.321020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.321081 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.321471 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.321475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.323022 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 21:10:09.323217 11661 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:10:09.323154 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.319209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:38989 (local address 127.27.117.126:35057)
0810 21:10:09.319248 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.319250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.319259 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:09.319314 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.319315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.319324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.319379 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.319607 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.320047 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.320680 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.320937 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.321017 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.323099 (+ 2082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:38989: BlockingRecv error: recv got EOF from 127.27.117.66:38989 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 21:10:09.394426 11668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:09.394624 11668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:09.394649 11668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:09.395785 11668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:09.395825 11668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:09.395850 11668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:10:09.397517 11668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/krb5kdc/oryx_127.27.117.67.keytab
--principal=oryx/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:33579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=39527
--tserver_master_addrs=127.27.117.126:35057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:09.397764 11668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:09.397994 11668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:09.403100 11680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:09.403177 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:09.403357 11668 server_base.cc:1034] running on GCE node
W20260810 21:10:09.403151 11679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:09.403616 11668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:09.403825 11668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:09.404956 11668 hybrid_clock.cc:648] HybridClock initialized: now 1786396209404950 us; error 35 us; skew 500 ppm
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396209, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:09.407516 11668 init.cc:377] Logged in from keytab as oryx/127.27.117.67@KRBTEST.COM (short username oryx)
I20260810 21:10:09.407977 11668 webserver.cc:460] Webserver started at http://127.27.117.67:39527/ using document root <none> and password file <none>
I20260810 21:10:09.408218 11668 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:09.408303 11668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:09.409694 11668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:09.410308 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.410526 11668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:09.410638 11668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "94fb67310b344653b0304a16cd142245"
format_stamp: "Formatted at 2026-08-10 21:10:08 on dist-test-slave-2kcd"
server_key: "0c0a17a283ad6ee1cf51f906f8f88f96"
server_key_iv: "1e472bd85b345c8f19ec06a320835db4"
server_key_version: "encryptionkey@0"
I20260810 21:10:09.410956 11668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:09.423024 11668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:09.425299 11668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:09.425470 11668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:09.425696 11668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:09.425994 11668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:09.426052 11668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.426096 11668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:09.426123 11668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:09.433614 11668 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:33579
I20260810 21:10:09.440853 11897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:33579 every 8 connection(s)
I20260810 21:10:09.441150 11668 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.448949 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 11668
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 21:10:09.450333 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.443011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45089 (local address 127.27.117.126:35057)
0810 21:10:09.443067 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.443069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.443301 (+ 232us) server_negotiation.cc:407] Connection header received
0810 21:10:09.444077 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.444078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.444092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.444151 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.445362 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.445873 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.446912 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.447125 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.447198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.450288 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:45089: BlockingRecv error: recv got EOF from 127.27.117.67:45089 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:10:09.450443 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.443003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:45089)
0810 21:10:09.443166 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.443210 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.443952 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.444223 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.444227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.445062 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.445258 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.445262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.445974 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.445975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.446803 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.446805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.447207 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.447208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.447345 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.447761 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.447768 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.450234 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
W20260810 21:10:09.450784 11899 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:35057 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.117.126:35057: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.455257 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.451000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:53747)
0810 21:10:09.451047 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.451057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.451096 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.451222 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.451223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.451346 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.451385 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.451386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.451976 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.451995 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.452554 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.452557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.453011 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.453012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.453067 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.453447 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.453452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.455187 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 21:10:09.455303 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.451054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53747 (local address 127.27.117.126:35057)
0810 21:10:09.451093 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.451095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.451104 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:09.451156 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.451157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.451165 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.451218 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.451460 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.451899 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.452645 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.452935 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.453012 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.455263 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:53747: BlockingRecv error: recv got EOF from 127.27.117.67:53747 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:10:09.457206 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.449495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.27.117.126:35057)
0810 21:10:09.449617 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.449619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.449628 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:09.449685 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.449685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.449695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.449752 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.450555 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.451154 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.451910 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.452142 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.452227 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.453550 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:09.453554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:09.453554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:09.453566 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:09.455664 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:09.456109 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:09.456112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:09.456113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:09.456219 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:09.456430 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:09.456431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:09.456432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:09.456662 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:09.456841 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:09.456970 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 21:10:09.457094 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:09 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:09.459703 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:09.455644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:41213)
0810 21:10:09.455694 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:09.455704 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:09.455742 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:09.455934 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:09.455934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:09.456057 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:09.456080 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.456081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.456661 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.456662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.457211 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:09.457213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.457575 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.457575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:09.457617 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.458056 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:09.458061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:09.459631 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 21:10:09.459827 11899 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:10:09.459749 11431 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:09.455734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:41213 (local address 127.27.117.126:35057)
0810 21:10:09.455775 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:09.455777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:09.455784 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:10:09.455809 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:09.455810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:09.455816 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:09.455915 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:09.456154 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.456581 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.457306 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:09.457509 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:09.457572 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:09.459694 (+ 2122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:41213: BlockingRecv error: recv got EOF from 127.27.117.67:41213 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 21:10:10 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:10 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:10.199944 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:10.195325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:49541)
0810 21:10:10.195473 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:10.195491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:10.195540 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:10.195675 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:10.195676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:10.195837 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:10.195889 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:10.195890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.196549 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.196550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:10.197153 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:10.197155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.197488 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.197488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:10.197526 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:10.197982 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:10.197986 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:10.199851 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260810 21:10:10.200021 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:10.195412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:49541 (local address 127.27.117.126:35057)
0810 21:10:10.195549 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:10.195551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:10.195559 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:10.195582 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:10.195582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:10.195591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:10.195652 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:10.195955 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.196480 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.197226 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.197421 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.197475 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:10.199943 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:49541: BlockingRecv error: recv got EOF from 127.27.117.65:49541 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
Aug 10 21:10:10 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:10 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:10.328295 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:10.323637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:47797)
0810 21:10:10.323781 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:10.323790 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:10.323844 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:10.323970 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:10.323971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:10.324152 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:10.324224 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:10.324226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.325127 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.325128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:10.325677 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:10.325679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.325994 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.325995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:10.326039 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:10.326510 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:10.326515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:10.328232 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260810 21:10:10.328425 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:10.323708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:47797 (local address 127.27.117.126:35057)
0810 21:10:10.323788 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:10.323790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:10.323847 (+ 57us) server_negotiation.cc:407] Connection header received
0810 21:10:10.323886 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:10.323887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:10.323898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:10.323952 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:10.324299 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.325062 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.325746 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.325922 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.325970 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:10.328345 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:47797: BlockingRecv error: recv got EOF from 127.27.117.66:47797 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 21:10:10 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:10 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:10.465173 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:10.460252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:45999)
0810 21:10:10.460455 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:10.460468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:10.460565 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:10.460736 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:10.460737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:10.460953 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:10.461001 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:10.461003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.461707 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.461709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:10.462243 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:10.462245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.462602 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.462603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:10.462667 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:10.463156 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:10.463161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:10.465067 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
W20260810 21:10:10.465207 11905 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:10.460374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45999 (local address 127.27.117.126:35057)
0810 21:10:10.460460 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:10.460462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:10.460551 (+ 89us) server_negotiation.cc:407] Connection header received
0810 21:10:10.460640 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:10.460641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:10.460656 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:10.460731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:10.461104 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.461603 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.462329 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:10.462517 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:10.462602 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:10.465143 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:45999: BlockingRecv error: recv got EOF from 127.27.117.67:45999 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 21:10:11 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:11 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:11.205422 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:11.200557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:37773)
0810 21:10:11.200749 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:11.200762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:11.200841 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:11.201082 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:11.201083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:11.201236 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:11.201287 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:11.201289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.202047 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.202048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:11.202580 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:11.202582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.202937 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.202938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:11.202980 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:11.203489 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:11.203494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:11.205358 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
W20260810 21:10:11.205549 11911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:11.200610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:37773 (local address 127.27.117.126:35057)
0810 21:10:11.200818 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:11.200820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:11.200830 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:10:11.200937 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:11.200938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:11.200957 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:11.200997 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:11.201342 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.201963 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.202662 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.202862 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.202941 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:11.205442 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:37773: BlockingRecv error: recv got EOF from 127.27.117.65:37773 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":136,"threads_started":1}
Aug 10 21:10:11 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:11 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:11.333732 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:11.328882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:49231)
0810 21:10:11.329029 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:11.329061 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:11.329128 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:11.329336 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:11.329337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:11.329518 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:11.329596 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:11.329598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.330415 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.330416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:11.330952 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:11.330954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.331277 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.331278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:11.331315 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:11.331725 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:11.331729 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:11.333672 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
W20260810 21:10:11.333864 11911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:11.329148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:49231 (local address 127.27.117.126:35057)
0810 21:10:11.329210 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:11.329212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:11.329228 (+ 16us) server_negotiation.cc:407] Connection header received
0810 21:10:11.329252 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:11.329253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:11.329262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:11.329312 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:11.329684 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.330349 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.331021 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.331216 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.331265 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:11.333784 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:49231: BlockingRecv error: recv got EOF from 127.27.117.66:49231 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 21:10:11 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:11 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:11.470077 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:11.465588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:33879)
0810 21:10:11.465756 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:11.465765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:11.465818 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:11.465951 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:11.465952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:11.466107 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:11.466154 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:11.466156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.466832 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.466832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:11.467321 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:11.467322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.467637 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.467638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:11.467677 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:11.468175 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:11.468180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:11.470020 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
W20260810 21:10:11.470240 11911 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:11.465705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:33879 (local address 127.27.117.126:35057)
0810 21:10:11.465755 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:11.465757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:11.465807 (+ 50us) server_negotiation.cc:407] Connection header received
0810 21:10:11.465873 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:11.465873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:11.465885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:11.465932 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:11.466254 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.466754 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.467396 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:11.467571 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:11.467627 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:11.470153 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:33879: BlockingRecv error: recv got EOF from 127.27.117.67:33879 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 21:10:12 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:12 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:12.210425 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:12.205844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:36939)
0810 21:10:12.205994 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:12.206008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:12.206082 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:12.206240 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:12.206240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:12.206416 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:12.206469 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:12.206471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.207160 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.207160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:12.207662 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:12.207663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.207988 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.207989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:12.208027 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:12.208471 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:12.208475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:12.210366 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260810 21:10:12.210595 11917 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:12.205944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:36939 (local address 127.27.117.126:35057)
0810 21:10:12.206103 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:12.206105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:12.206114 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:12.206135 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:12.206136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:12.206158 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:12.206224 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:12.206567 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.207079 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.207729 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.207927 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.207975 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:12.210480 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:36939: BlockingRecv error: recv got EOF from 127.27.117.65:36939 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
Aug 10 21:10:12 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:12 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:12.338541 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:12.334203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:47147)
0810 21:10:12.334372 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:12.334382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:12.334443 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:12.334563 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:12.334563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:12.334728 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:12.334778 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:12.334779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.335383 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.335384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:12.335901 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:12.335903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.336211 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.336212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:12.336249 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:12.336678 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:12.336682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:12.338478 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
W20260810 21:10:12.338681 11917 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:12.334258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:47147 (local address 127.27.117.126:35057)
0810 21:10:12.334312 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:12.334314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:12.334431 (+ 117us) server_negotiation.cc:407] Connection header received
0810 21:10:12.334497 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:12.334498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:12.334509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:12.334545 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:12.334838 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.335307 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.335973 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.336149 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.336198 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:12.338598 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:47147: BlockingRecv error: recv got EOF from 127.27.117.66:47147 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 21:10:12 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:12 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:12.475157 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:12.470733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:41637)
0810 21:10:12.470926 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:12.470941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:12.471005 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:12.471194 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:12.471195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:12.471367 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:12.471419 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:12.471420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.472032 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.472033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:12.472537 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:12.472538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.472852 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.472853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:12.472892 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:12.473399 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:12.473404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:12.475100 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":107,"threads_started":1}
W20260810 21:10:12.475255 11917 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:12.470887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:41637 (local address 127.27.117.126:35057)
0810 21:10:12.470960 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:12.470962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:12.470987 (+ 25us) server_negotiation.cc:407] Connection header received
0810 21:10:12.471103 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:12.471104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:12.471119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:12.471174 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:12.471489 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.471957 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.472609 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:12.472796 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:12.472844 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:12.475188 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:41637: BlockingRecv error: recv got EOF from 127.27.117.67:41637 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 21:10:13 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:13 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:13.215745 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:13.210963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:48549)
0810 21:10:13.211098 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:13.211111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:13.211165 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:13.211301 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:13.211301 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:13.211460 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:13.211513 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:13.211515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.212156 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.212158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:13.212704 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:13.212706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.213108 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.213109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:13.213166 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:13.213712 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:13.213718 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:13.215688 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260810 21:10:13.215924 11924 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:13.211028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:48549 (local address 127.27.117.126:35057)
0810 21:10:13.211177 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:13.211179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:13.211205 (+ 26us) server_negotiation.cc:407] Connection header received
0810 21:10:13.211225 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:13.211226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:13.211236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:13.211285 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:13.211577 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.212053 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.212774 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.213018 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.213075 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:13.215809 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:48549: BlockingRecv error: recv got EOF from 127.27.117.65:48549 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
Aug 10 21:10:13 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:13 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:13.348542 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:13.339097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:59103)
0810 21:10:13.339311 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:13.339335 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:13.339432 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:13.339609 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:13.339610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:13.339808 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:13.339858 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:13.339860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.340552 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.340553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:13.341186 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:13.341188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.341553 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.341555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:13.341617 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:13.346674 (+ 5057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:13.346679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:13.348473 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
W20260810 21:10:13.348698 11924 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:13.339199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:59103 (local address 127.27.117.126:35057)
0810 21:10:13.339315 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:13.339318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:13.339410 (+ 92us) server_negotiation.cc:407] Connection header received
0810 21:10:13.339508 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:13.339509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:13.339524 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:13.339599 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:13.339948 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.340472 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.341272 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.341469 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.341542 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:13.348616 (+ 7074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:59103: BlockingRecv error: recv got EOF from 127.27.117.66:59103 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 10 21:10:13 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:13 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:13.480228 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:13.475643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:36401)
0810 21:10:13.475829 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:13.475841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:13.475904 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:13.476029 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:13.476030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:13.476204 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:13.476255 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:13.476256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.476919 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.476920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:13.477440 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:13.477442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.477876 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.477877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:13.477918 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:13.478354 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:13.478358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:13.480169 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260810 21:10:13.480300 11924 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:13.475706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:36401 (local address 127.27.117.126:35057)
0810 21:10:13.475759 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:13.475761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:13.475888 (+ 127us) server_negotiation.cc:407] Connection header received
0810 21:10:13.475956 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:13.475956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:13.475969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:13.476008 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:13.476321 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.476824 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.477608 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:13.477801 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:13.477888 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:13.480252 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:36401: BlockingRecv error: recv got EOF from 127.27.117.67:36401 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 21:10:14 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:14 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:14.221115 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:14.216264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:45573)
0810 21:10:14.216420 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:14.216430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:14.216486 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:14.216612 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:14.216613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:14.216798 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:14.216852 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:14.216853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.217486 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.217487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:14.218037 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:14.218038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.218587 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.218589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:14.218662 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:14.219156 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:14.219160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:14.221019 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
W20260810 21:10:14.221190 11928 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:14.216347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:45573 (local address 127.27.117.126:35057)
0810 21:10:14.216500 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:14.216501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:14.216509 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:14.216530 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:14.216530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:14.216541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:14.216596 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:14.216918 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.217421 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.218136 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.218477 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.218549 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:14.221102 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:45573: BlockingRecv error: recv got EOF from 127.27.117.65:45573 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
Aug 10 21:10:14 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:14 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:14.353334 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:14.349019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:37415)
0810 21:10:14.349165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:14.349175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:14.349228 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:14.349344 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:14.349344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:14.349513 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:14.349566 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:14.349567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.350226 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.350226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:14.350742 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:14.350744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.351088 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.351089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:14.351130 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:14.351568 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:14.351573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:14.353239 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
W20260810 21:10:14.353381 11928 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:14.349143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:37415 (local address 127.27.117.126:35057)
0810 21:10:14.349211 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:14.349213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:14.349230 (+ 17us) server_negotiation.cc:407] Connection header received
0810 21:10:14.349288 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:14.349289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:14.349299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:14.349338 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:14.349627 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.350155 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.350815 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.351022 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.351075 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:14.353319 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:37415: BlockingRecv error: recv got EOF from 127.27.117.66:37415 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 21:10:14 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:14 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:14.485136 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:14.480905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:38469)
0810 21:10:14.481034 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:14.481044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:14.481099 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:14.481208 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:14.481209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:14.481359 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:14.481416 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:14.481418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.482033 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.482034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:14.482551 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:14.482552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.482853 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.482854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:14.482892 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:14.483370 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:14.483375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:14.485078 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
W20260810 21:10:14.485226 11928 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:14.480965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:38469 (local address 127.27.117.126:35057)
0810 21:10:14.481024 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:14.481026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:14.481097 (+ 71us) server_negotiation.cc:407] Connection header received
0810 21:10:14.481140 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:14.481141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:14.481153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:14.481198 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:14.481468 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.481968 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.482616 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:14.482793 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:14.482842 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:14.485161 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:38469: BlockingRecv error: recv got EOF from 127.27.117.67:38469 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 21:10:15 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:15 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:15.226112 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:15.221632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:44705)
0810 21:10:15.221792 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:15.221807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:15.221875 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:15.222013 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:15.222014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:15.222184 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:15.222235 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:15.222237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.222876 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.222877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:15.223455 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:15.223457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.223796 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.223797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:15.223836 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:15.224271 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:15.224275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:15.226052 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
W20260810 21:10:15.226229 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:15.221676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:44705 (local address 127.27.117.126:35057)
0810 21:10:15.221811 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:15.221812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:15.221858 (+ 46us) server_negotiation.cc:407] Connection header received
0810 21:10:15.221932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:15.221933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:15.221951 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:15.221999 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:15.222300 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.222804 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.223531 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.223722 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.223774 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:15.226149 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:44705: BlockingRecv error: recv got EOF from 127.27.117.65:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
Aug 10 21:10:15 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:15 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:15.358376 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:15.353833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:48465)
0810 21:10:15.353980 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:15.353991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:15.354058 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:15.354228 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:15.354229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:15.354418 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:15.354466 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:15.354468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.355131 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.355132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:15.355659 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:15.355660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.356019 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.356020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:15.356084 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:15.356524 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:15.356528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:15.358283 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260810 21:10:15.358438 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:15.354016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:48465 (local address 127.27.117.126:35057)
0810 21:10:15.354088 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:15.354089 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:15.354097 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:15.354122 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:15.354123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:15.354140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:15.354219 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:15.354547 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.355048 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.355749 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.355946 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.356021 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:15.358373 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:48465: BlockingRecv error: recv got EOF from 127.27.117.66:48465 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 21:10:15 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:15 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:15.490491 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:15.485702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:52407)
0810 21:10:15.485901 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:15.485911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:15.485996 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:15.486156 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:15.486157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:15.486356 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:15.486405 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:15.486407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.487114 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.487116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:15.487650 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:15.487652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.488012 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.488014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:15.488089 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:15.488581 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:15.488585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:15.490392 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
W20260810 21:10:15.490556 11932 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:15.485762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:52407 (local address 127.27.117.126:35057)
0810 21:10:15.485866 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:15.485869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:15.485978 (+ 109us) server_negotiation.cc:407] Connection header received
0810 21:10:15.486059 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:15.486060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:15.486074 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:15.486150 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:15.486488 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.487010 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.487739 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:15.487933 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:15.488013 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:15.490478 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:52407: BlockingRecv error: recv got EOF from 127.27.117.67:52407 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 21:10:16 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:16 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:16.231346 11935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:16.226599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:45107)
0810 21:10:16.226744 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:16.226755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:16.226815 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:16.226963 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:16.226964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:16.227200 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:16.227283 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:16.227286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.227983 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.227984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:16.228520 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:16.228522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.228984 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.228985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:16.229046 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:16.229481 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:16.229485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:16.231288 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260810 21:10:16.231490 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:16.226675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:45107 (local address 127.27.117.126:35057)
0810 21:10:16.226812 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:16.226813 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:16.226822 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:16.226863 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:16.226864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:16.226891 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:16.226956 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:16.227379 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.227891 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.228638 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.228896 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.228980 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:16.231382 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:45107: BlockingRecv error: recv got EOF from 127.27.117.65:45107 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
Aug 10 21:10:16 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:16 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:16.363811 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:16.359004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:42665)
0810 21:10:16.359207 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:16.359225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:16.359313 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:16.359504 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:16.359506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:16.359695 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:16.359746 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:16.359748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.360444 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.360445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:16.361028 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:16.361030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.361419 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.361420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:16.361486 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:16.361954 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:16.361958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:16.363714 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
W20260810 21:10:16.363866 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:16.359067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:42665 (local address 127.27.117.126:35057)
0810 21:10:16.359165 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:16.359167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:16.359296 (+ 129us) server_negotiation.cc:407] Connection header received
0810 21:10:16.359405 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:16.359406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:16.359426 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:16.359496 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:16.359830 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.360348 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.361123 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.361332 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.361419 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:16.363803 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:42665: BlockingRecv error: recv got EOF from 127.27.117.66:42665 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 21:10:16 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:16 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:16.495913 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:16.491016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:56745)
0810 21:10:16.491197 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:16.491217 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:16.491327 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:16.491478 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:16.491479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:16.491663 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:16.491712 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:16.491714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.492406 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.492407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:16.493014 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:16.493017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.493400 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.493401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:16.493465 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:16.493972 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:16.493977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:16.495851 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
W20260810 21:10:16.496055 11936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:16.491142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:56745 (local address 127.27.117.126:35057)
0810 21:10:16.491233 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:16.491235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:16.491299 (+ 64us) server_negotiation.cc:407] Connection header received
0810 21:10:16.491405 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:16.491406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:16.491421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:16.491474 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:16.491790 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.492298 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.493105 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:16.493304 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:16.493398 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:16.495978 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:56745: BlockingRecv error: recv got EOF from 127.27.117.67:56745 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 21:10:17 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:17 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:17.236477 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:17.231761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:47975)
0810 21:10:17.231896 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:17.231906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:17.231953 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:17.232170 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:17.232170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:17.232331 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:17.232383 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:17.232385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.233156 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.233157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:17.233659 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:17.233661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.234029 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.234030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:17.234078 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:17.234619 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:17.234623 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:17.236419 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
W20260810 21:10:17.236681 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:17.231903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:47975 (local address 127.27.117.126:35057)
0810 21:10:17.232037 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:17.232039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:17.232048 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:17.232079 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:17.232080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:17.232092 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:17.232156 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:17.232532 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.233082 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.233737 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.233957 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.234018 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:17.236596 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:47975: BlockingRecv error: recv got EOF from 127.27.117.65:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
Aug 10 21:10:17 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:17 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:17.369036 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:17.364367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:36845)
0810 21:10:17.364531 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:17.364541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:17.364605 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:17.364739 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:17.364739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:17.364922 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:17.364973 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:17.364974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.365714 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.365714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:17.366244 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:17.366245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.366565 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.366566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:17.366604 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:17.367031 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:17.367035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:17.368924 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
W20260810 21:10:17.369117 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:17.364452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:36845 (local address 127.27.117.126:35057)
0810 21:10:17.364536 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:17.364539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:17.364604 (+ 65us) server_negotiation.cc:407] Connection header received
0810 21:10:17.364662 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:17.364663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:17.364674 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:17.364716 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:17.365042 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.365645 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.366311 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.366486 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.366553 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:17.369046 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:36845: BlockingRecv error: recv got EOF from 127.27.117.66:36845 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 21:10:17 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:17 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:17.501225 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:17.496486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:54239)
0810 21:10:17.496661 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:17.496672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:17.496731 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:17.496956 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:17.496957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:17.497199 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:17.497261 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:17.497263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.497932 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.497933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:17.498430 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:17.498432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.498743 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.498743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:17.498781 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:17.499214 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:17.499218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:17.501158 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20260810 21:10:17.501338 11941 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:17.496544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:54239 (local address 127.27.117.126:35057)
0810 21:10:17.496619 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:17.496622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:17.496735 (+ 113us) server_negotiation.cc:407] Connection header received
0810 21:10:17.496773 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:17.496773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:17.496820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:17.496923 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:17.497342 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.497837 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.498504 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:17.498678 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:17.498729 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:17.501247 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:54239: BlockingRecv error: recv got EOF from 127.27.117.67:54239 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 21:10:18 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:18 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:18.241592 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:18.237028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:50425)
0810 21:10:18.237180 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:18.237195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:18.237263 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:18.237412 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:18.237413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:18.237582 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:18.237628 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:18.237630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.238263 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.238264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:18.238789 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:18.238794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.239118 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.239119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:18.239164 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:18.239592 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:18.239596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:18.241528 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
W20260810 21:10:18.241710 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:18.237135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:50425 (local address 127.27.117.126:35057)
0810 21:10:18.237296 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:18.237297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:18.237305 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:18.237325 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:18.237326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:18.237336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:18.237393 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:18.237697 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.238189 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.238863 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.239053 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.239102 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:18.241628 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:50425: BlockingRecv error: recv got EOF from 127.27.117.65:50425 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
Aug 10 21:10:18 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:18 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:18.377328 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:18.369495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:60595)
0810 21:10:18.369654 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:18.369665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:18.369722 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:18.369881 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:18.369882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:18.370056 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:18.370108 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:18.370110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.370726 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.370727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:18.371252 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:18.371254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.371644 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.371645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:18.371724 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:18.375394 (+ 3670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:18.375399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:18.377261 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
W20260810 21:10:18.377400 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:18.369606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:60595 (local address 127.27.117.126:35057)
0810 21:10:18.369669 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:18.369671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:18.369712 (+ 41us) server_negotiation.cc:407] Connection header received
0810 21:10:18.369782 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:18.369783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:18.369795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:18.369838 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:18.370174 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.370659 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.371320 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.371551 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.371630 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:18.377353 (+ 5723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:60595: BlockingRecv error: recv got EOF from 127.27.117.66:60595 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 21:10:18 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:18 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:18.506546 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:18.501660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:58277)
0810 21:10:18.501832 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:18.501849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:18.501939 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:18.502100 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:18.502101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:18.502305 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:18.502374 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:18.502377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.503042 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.503044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:18.503586 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:18.503588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.503954 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.503955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:18.504013 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:18.504514 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:18.504519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:18.506441 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
W20260810 21:10:18.506577 11948 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:18.501727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:58277 (local address 127.27.117.126:35057)
0810 21:10:18.501804 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:18.501806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:18.501917 (+ 111us) server_negotiation.cc:407] Connection header received
0810 21:10:18.502000 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:18.502001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:18.502014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:18.502095 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:18.502461 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.502956 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.503677 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:18.503877 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:18.503955 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:18.506516 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:58277: BlockingRecv error: recv got EOF from 127.27.117.67:58277 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 10 21:10:19 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:19 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:19.247167 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:19.242227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:56647)
0810 21:10:19.242385 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:19.242396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:19.242452 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:19.242678 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:19.242678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:19.242840 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:19.242892 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:19.242894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.243523 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.243524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:19.244155 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:19.244157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.244515 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.244516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:19.244581 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:19.245160 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:19.245166 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:19.247110 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
W20260810 21:10:19.247295 11953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:19.242430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:56647 (local address 127.27.117.126:35057)
0810 21:10:19.242571 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:19.242573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:19.242580 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:10:19.242600 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:19.242601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:19.242611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:19.242672 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:19.242955 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.243455 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.244221 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.244432 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.244498 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:19.247200 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:56647: BlockingRecv error: recv got EOF from 127.27.117.65:56647 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
Aug 10 21:10:19 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:19 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:19.382382 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:19.377890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:34565)
0810 21:10:19.378060 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:19.378069 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:19.378127 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:19.378256 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:19.378257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:19.378430 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:19.378479 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:19.378481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.379102 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.379103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:19.379616 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:19.379617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.379937 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.379938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:19.379979 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:19.380411 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:19.380415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:19.382270 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
W20260810 21:10:19.382428 11953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:19.377977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:34565 (local address 127.27.117.126:35057)
0810 21:10:19.378044 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:19.378046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:19.378129 (+ 83us) server_negotiation.cc:407] Connection header received
0810 21:10:19.378186 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:19.378187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:19.378198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:19.378239 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:19.378545 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.379032 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.379688 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.379873 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.379923 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:19.382361 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:34565: BlockingRecv error: recv got EOF from 127.27.117.66:34565 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 21:10:19 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:19 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:19.511870 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:19.507102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:38813)
0810 21:10:19.507280 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:19.507296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:19.507389 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:19.507588 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:19.507589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:19.507780 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:19.507828 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:19.507830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.508502 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.508503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:19.509112 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:19.509115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.509482 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.509483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:19.509550 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:19.509983 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:19.509988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:19.511813 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
W20260810 21:10:19.511986 11953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:19.507244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:38813 (local address 127.27.117.126:35057)
0810 21:10:19.507305 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:19.507307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:19.507368 (+ 61us) server_negotiation.cc:407] Connection header received
0810 21:10:19.507470 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:19.507471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:19.507488 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:19.507575 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:19.507909 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.508409 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.509209 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:19.509400 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:19.509483 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:19.511909 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:38813: BlockingRecv error: recv got EOF from 127.27.117.67:38813 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 21:10:20 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:20 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:20.253242 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:20.247664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:43073)
0810 21:10:20.247809 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:20.247827 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:20.247878 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:20.248026 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:20.248027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:20.248207 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:20.248259 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:20.248261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.248927 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.248927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:20.249527 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:20.249531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.250098 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.250099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:20.250172 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:20.250723 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:20.250729 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:20.253151 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
W20260810 21:10:20.253504 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:20.247738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:43073 (local address 127.27.117.126:35057)
0810 21:10:20.247884 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:20.247891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:20.247899 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:20.247934 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:20.247934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:20.247945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:20.248008 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:20.248328 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.248858 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.249638 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.249995 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.250077 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:20.253440 (+ 3363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:43073: BlockingRecv error: recv got EOF from 127.27.117.65:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
Aug 10 21:10:20 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:20 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:20.387538 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:20.382924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:59629)
0810 21:10:20.383056 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:20.383067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:20.383125 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:20.383272 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:20.383273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:20.383441 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:20.383488 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:20.383490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.384104 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.384105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:20.384648 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:20.384650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.385023 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.385024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:20.385066 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:20.385580 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:20.385585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:20.387436 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260810 21:10:20.387570 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:20.383042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:59629 (local address 127.27.117.126:35057)
0810 21:10:20.383120 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:20.383122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:20.383130 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:20.383182 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:20.383183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:20.383194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:20.383236 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:20.383555 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.384038 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.384720 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.384960 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.385012 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:20.387510 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:59629: BlockingRecv error: recv got EOF from 127.27.117.66:59629 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 21:10:20 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:20 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:20.517314 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:20.512453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:46923)
0810 21:10:20.512610 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:20.512620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:20.512674 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:20.512802 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:20.512803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:20.513011 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:20.513086 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:20.513092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.513757 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.513758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:20.514371 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:20.514373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.514756 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.514757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:20.514831 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:20.515281 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:20.515286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:20.517224 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
W20260810 21:10:20.517376 11960 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:20.512510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:46923 (local address 127.27.117.126:35057)
0810 21:10:20.512600 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:20.512601 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:20.512662 (+ 61us) server_negotiation.cc:407] Connection header received
0810 21:10:20.512717 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:20.512717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:20.512729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:20.512814 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:20.513154 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.513678 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.514448 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:20.514633 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:20.514683 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:20.517307 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:46923: BlockingRecv error: recv got EOF from 127.27.117.67:46923 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 21:10:21 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:21 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:21.258519 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:21.253752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:36005)
0810 21:10:21.253909 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:21.253920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:21.253982 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:21.254160 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:21.254161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:21.254350 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:21.254401 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:21.254404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.255191 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.255192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:21.255730 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:21.255732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.256056 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.256058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:21.256104 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:21.256528 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:21.256533 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:21.258425 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
W20260810 21:10:21.258602 11970 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:21.253856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:36005 (local address 127.27.117.126:35057)
0810 21:10:21.253980 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:21.253982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:21.253990 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:21.254040 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:21.254040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:21.254050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:21.254107 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:21.254498 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.255122 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.255800 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.255989 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.256036 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:21.258517 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:36005: BlockingRecv error: recv got EOF from 127.27.117.65:36005 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":54,"threads_started":1}
Aug 10 21:10:21 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:21 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:21.392925 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:21.388008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:58277)
0810 21:10:21.388167 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:21.388179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:21.388253 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:21.388509 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:21.388510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:21.388822 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:21.388889 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:21.388892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.389671 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.389672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:21.390213 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:21.390215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.390531 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.390532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:21.390572 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:21.391088 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:21.391093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:21.392859 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
W20260810 21:10:21.392969 11970 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:21.388064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:58277 (local address 127.27.117.126:35057)
0810 21:10:21.388125 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:21.388127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:21.388233 (+ 106us) server_negotiation.cc:407] Connection header received
0810 21:10:21.388332 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:21.388333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:21.388359 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:21.388475 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:21.388991 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.389596 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.390285 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.390467 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.390518 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:21.392921 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:58277: BlockingRecv error: recv got EOF from 127.27.117.66:58277 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 10 21:10:21 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:21 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 21:10:21.522691 11970 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:21.517917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45359 (local address 127.27.117.126:35057)
0810 21:10:21.518022 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:21.518025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:21.518135 (+ 110us) server_negotiation.cc:407] Connection header received
0810 21:10:21.518220 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:21.518221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:21.518237 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:21.518329 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:21.518668 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.519178 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.519879 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.520085 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.520173 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:21.522629 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:45359: BlockingRecv error: recv got EOF from 127.27.117.67:45359 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 21:10:21.522746 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:21.517773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:45359)
0810 21:10:21.518047 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:21.518058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:21.518155 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:21.518329 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:21.518330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:21.518530 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:21.518574 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:21.518576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.519261 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.519262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:21.519790 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:21.519791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:21.520176 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:21.520177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:21.520243 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:21.520700 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:21.520705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:21.522551 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":159,"threads_started":1}
Aug 10 21:10:22 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:22 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:22.263554 11975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:22.259001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:52057)
0810 21:10:22.259144 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:22.259156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:22.259216 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:22.259406 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:22.259406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:22.259581 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:22.259633 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:22.259634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.260274 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.260274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:22.260845 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:22.260847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.261193 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.261194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:22.261234 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:22.261659 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:22.261664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:22.263450 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260810 21:10:22.263628 11979 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:22.259177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:52057 (local address 127.27.117.126:35057)
0810 21:10:22.259281 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:22.259283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:22.259290 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:10:22.259313 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:22.259313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:22.259323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:22.259389 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:22.259706 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.260205 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.260926 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.261127 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.261181 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:22.263545 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:52057: BlockingRecv error: recv got EOF from 127.27.117.65:52057 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
Aug 10 21:10:22 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:22 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:22.397774 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:22.393435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:37979)
0810 21:10:22.393597 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:22.393610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:22.393682 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:22.393789 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:22.393790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:22.393959 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:22.394006 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:22.394007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.394656 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.394657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:22.395180 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:22.395181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.395510 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.395511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:22.395560 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:22.395971 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:22.395976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:22.397701 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
W20260810 21:10:22.397845 11979 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:22.393565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:37979 (local address 127.27.117.126:35057)
0810 21:10:22.393638 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:22.393640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:22.393668 (+ 28us) server_negotiation.cc:407] Connection header received
0810 21:10:22.393727 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:22.393728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:22.393738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:22.393786 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:22.394074 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.394587 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.395250 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.395447 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.395496 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:22.397798 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:37979: BlockingRecv error: recv got EOF from 127.27.117.66:37979 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 21:10:22 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:22 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:22.528195 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:22.523196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:42183)
0810 21:10:22.523476 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:22.523488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:22.523584 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:22.523741 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:22.523742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:22.523931 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:22.523976 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:22.523979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.524903 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.524904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:22.525491 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:22.525493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.525840 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.525841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:22.525903 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:22.526384 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:22.526389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:22.528128 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":205,"threads_started":1}
W20260810 21:10:22.528316 11979 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:22.523385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:42183 (local address 127.27.117.126:35057)
0810 21:10:22.523461 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:22.523463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:22.523568 (+ 105us) server_negotiation.cc:407] Connection header received
0810 21:10:22.523648 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:22.523650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:22.523663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:22.523735 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:22.524105 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.524709 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.525578 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:22.525759 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:22.525815 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:22.528218 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:42183: BlockingRecv error: recv got EOF from 127.27.117.67:42183 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 21:10:23 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:23 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:23.269032 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:23.264198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:58963)
0810 21:10:23.264414 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:23.264426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:23.264491 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:23.264735 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:23.264736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:23.265009 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:23.265086 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:23.265089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.265820 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.265820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:23.266371 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:23.266373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.266685 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.266686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:23.266725 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:23.267183 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:23.267188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:23.268961 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
W20260810 21:10:23.269136 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:23.264314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:58963 (local address 127.27.117.126:35057)
0810 21:10:23.264431 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:23.264433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:23.264566 (+ 133us) server_negotiation.cc:407] Connection header received
0810 21:10:23.264597 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:23.264598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:23.264613 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:23.264689 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:23.265178 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.265750 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.266444 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.266625 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.266673 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:23.269060 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:58963: BlockingRecv error: recv got EOF from 127.27.117.65:58963 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
Aug 10 21:10:23 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:23 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:23.412431 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:23.398357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:49129)
0810 21:10:23.398561 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:23.398571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:23.398628 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:23.398799 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:23.398800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:23.398975 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:23.399025 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:23.399026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.399662 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.399663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:23.400204 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:23.400205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.400522 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.400523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:23.400568 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:23.410540 (+ 9972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:23.410546 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:23.412367 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":116,"threads_started":1}
W20260810 21:10:23.412593 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:23.398453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:49129 (local address 127.27.117.126:35057)
0810 21:10:23.398515 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:23.398516 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:23.398624 (+ 108us) server_negotiation.cc:407] Connection header received
0810 21:10:23.398691 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:23.398692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:23.398702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:23.398772 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:23.399084 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.399586 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.400268 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.400459 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.400507 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:23.412522 (+ 12015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:49129: BlockingRecv error: recv got EOF from 127.27.117.66:49129 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 21:10:23 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:23 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.67@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:23.533694 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:23.528639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.67:53201)
0810 21:10:23.528833 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:23.528845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:23.528937 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:23.529108 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:23.529109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:23.529299 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:23.529345 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:23.529347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.530030 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.530031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:23.530562 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:23.530564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.530910 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.530911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:23.530969 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:23.531491 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:23.531496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:23.533636 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
W20260810 21:10:23.533804 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:23.528718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53201 (local address 127.27.117.126:35057)
0810 21:10:23.528814 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:23.528816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:23.528916 (+ 100us) server_negotiation.cc:407] Connection header received
0810 21:10:23.529016 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:23.529017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:23.529029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:23.529087 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:23.529434 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.529935 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.530645 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:23.530835 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:23.530912 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:23.533731 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.67:53201: BlockingRecv error: recv got EOF from 127.27.117.67:53201 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 21:10:24 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:24 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.65@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:24.274003 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.269552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.65:56903)
0810 21:10:24.269674 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:24.269688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:24.269751 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:24.269864 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:24.269865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:24.270032 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:24.270084 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.270085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.270716 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.270716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.271256 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.271257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.271599 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.271599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.271639 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.272160 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:24.272164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:24.273944 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260810 21:10:24.274127 11993 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:24.269614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:56903 (local address 127.27.117.126:35057)
0810 21:10:24.269729 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:24.269731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:24.269739 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:10:24.269796 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:24.269796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:24.269807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:24.269861 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:24.270144 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.270644 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.271329 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.271524 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.271583 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.274039 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.65:56903: BlockingRecv error: recv got EOF from 127.27.117.65:56903 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
Aug 10 21:10:24 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 21:10:24 dist-test-slave-2kcd krb5kdc[10108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.117.66@KRBTEST.COM for oryx/127.27.117.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 21:10:24.417582 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.413043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35057 (local address 127.27.117.66:53949)
0810 21:10:24.413168 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:24.413180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:24.413245 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:24.413439 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:24.413440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:24.413615 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:24.413662 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.413663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.414311 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.414312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.414834 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.414836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.415226 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.415227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.415286 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.415704 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:24.415708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:24.417478 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.117.126:35057: Server oryx/127.27.117.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
W20260810 21:10:24.417636 11993 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:24.413121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:53949 (local address 127.27.117.126:35057)
0810 21:10:24.413168 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:24.413170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:24.413229 (+ 59us) server_negotiation.cc:407] Connection header received
0810 21:10:24.413333 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:24.413334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:24.413344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:24.413430 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:24.413744 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.414222 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.414920 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.415120 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.415225 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.417572 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.117.66:53949: BlockingRecv error: recv got EOF from 127.27.117.66:53949 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 21:10:24.451483 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 11145
I20260810 21:10:24.466202 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 11432
I20260810 21:10:24.480073 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 11668
I20260810 21:10:24.492447 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 11037
2026-08-10T21:10:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17778 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T21:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:24Z Disabled control of system clock
I20260810 21:10:24.529242 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38289
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:41679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 21:10:24.610320 12000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:24.610581 12000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:24.610615 12000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:24.611760 12000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:24.611833 12000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:24.611855 12000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:24.611886 12000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:24.611928 12000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:24.613459 12000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:41679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:24.613747 12000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:24.614055 12000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:24.619230 12009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:24.619354 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:24.619549 12000 server_base.cc:1034] running on GCE node
W20260810 21:10:24.619426 12011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:24.619859 12000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:24.620189 12000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:24.621354 12000 hybrid_clock.cc:648] HybridClock initialized: now 1786396224621335 us; error 38 us; skew 500 ppm
I20260810 21:10:24.622578 12000 webserver.cc:460] Webserver started at http://127.27.117.126:42535/ using document root <none> and password file <none>
I20260810 21:10:24.622838 12000 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:24.622907 12000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:24.623062 12000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:24.624200 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "ceb1c1b878a84992a7f28e7d38b94057"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "bd9bb5afcd7ccfa199b4a4f96ad32ba7"
server_key_iv: "fecf8c9240462d5177b3287c1cb1f1a4"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.624567 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "ceb1c1b878a84992a7f28e7d38b94057"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "bd9bb5afcd7ccfa199b4a4f96ad32ba7"
server_key_iv: "fecf8c9240462d5177b3287c1cb1f1a4"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.625924 12000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:10:24.626750 12018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.626953 12000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:24.627048 12000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "ceb1c1b878a84992a7f28e7d38b94057"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "bd9bb5afcd7ccfa199b4a4f96ad32ba7"
server_key_iv: "fecf8c9240462d5177b3287c1cb1f1a4"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.627125 12000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:24.635810 12000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:24.636099 12000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:24.636245 12000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:24.640347 12000 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:38289
I20260810 21:10:24.640380 12106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:38289 every 8 connection(s)
I20260810 21:10:24.640722 12000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:10:24.641446 12111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:24.643476 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 12000
I20260810 21:10:24.643589 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:10:24.643780 28117 external_mini_cluster.cc:1428] Setting key 97b19f85e756e58bb39e8ed340f9018d
I20260810 21:10:24.644520 12111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057: Bootstrap starting.
I20260810 21:10:24.645231 12111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:24.645625 12111 log.cc:824] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:24.646309 12111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057: No bootstrap required, opened a new log
I20260810 21:10:24.648823 12111 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb1c1b878a84992a7f28e7d38b94057" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38289 } }
I20260810 21:10:24.648993 12111 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:24.649019 12111 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb1c1b878a84992a7f28e7d38b94057, State: Initialized, Role: FOLLOWER
I20260810 21:10:24.649158 12111 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb1c1b878a84992a7f28e7d38b94057" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38289 } }
I20260810 21:10:24.649276 12111 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:10:24.649317 12111 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:10:24.649375 12111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:24.649756 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.644354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.27.117.126:38289)
0810 21:10:24.644551 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:24.644559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:24.644601 (+ 42us) server_negotiation.cc:407] Connection header received
0810 21:10:24.645096 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:24.645103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:24.645164 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:24.645279 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:24.645545 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.646488 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.647592 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.647992 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.648078 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.648883 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:24.648887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:24.648887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:24.649121 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:24.649183 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:24.649197 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:24.649262 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:24.649543 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 21:10:24.649604 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260810 21:10:24.649972 12111 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb1c1b878a84992a7f28e7d38b94057" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38289 } }
I20260810 21:10:24.650085 12111 leader_election.cc:302] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceb1c1b878a84992a7f28e7d38b94057; no voters:
I20260810 21:10:24.650244 12111 leader_election.cc:288] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:10:24.650311 12118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:24.650482 12111 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:10:24.650519 12118 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [term 1 LEADER]: Becoming Leader. State: Replica: ceb1c1b878a84992a7f28e7d38b94057, State: Running, Role: LEADER
I20260810 21:10:24.650669 12118 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb1c1b878a84992a7f28e7d38b94057" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38289 } }
I20260810 21:10:24.651032 12123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb1c1b878a84992a7f28e7d38b94057. Latest consensus state: current_term: 1 leader_uuid: "ceb1c1b878a84992a7f28e7d38b94057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb1c1b878a84992a7f28e7d38b94057" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38289 } } }
I20260810 21:10:24.651022 12119 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceb1c1b878a84992a7f28e7d38b94057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb1c1b878a84992a7f28e7d38b94057" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38289 } } }
I20260810 21:10:24.651124 12123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:24.651129 12119 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:24.651331 12136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:24.651831 12136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:10:24.653306 12136 catalog_manager.cc:1347] Generated new cluster ID: 20c44be2e8734447a83c6ab7bc95a847
I20260810 21:10:24.653359 12136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:10:24.668982 12136 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:10:24.669550 12136 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:24.677567 12136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceb1c1b878a84992a7f28e7d38b94057: Generated new TSK 0
I20260810 21:10:24.677747 12136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:10:24.683451 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38289
--builtin_ntp_servers=127.27.117.84:41679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 21:10:24.765260 12149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:24.765503 12149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:24.765542 12149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:24.766654 12149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:24.766714 12149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:24.766753 12149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:10:24.768038 12149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:41679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:24.768311 12149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:24.768639 12149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:24.773622 12163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:24.773628 12167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:24.773695 12164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:24.773847 12149 server_base.cc:1034] running on GCE node
I20260810 21:10:24.774066 12149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:24.774305 12149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:24.775454 12149 hybrid_clock.cc:648] HybridClock initialized: now 1786396224775443 us; error 29 us; skew 500 ppm
I20260810 21:10:24.776620 12149 webserver.cc:460] Webserver started at http://127.27.117.65:36677/ using document root <none> and password file <none>
I20260810 21:10:24.776918 12149 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:24.776990 12149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:24.777127 12149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:24.778151 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "448dec23cb0a4013b62da60624c4fd18"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "e51e7948e9fe0f3ae30386f3c491758a"
server_key_iv: "e75cfae44953e665e44d3de4591d5b36"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.778527 12149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "448dec23cb0a4013b62da60624c4fd18"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "e51e7948e9fe0f3ae30386f3c491758a"
server_key_iv: "e75cfae44953e665e44d3de4591d5b36"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.779814 12149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:10:24.780525 12177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.780723 12149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:24.780851 12149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "448dec23cb0a4013b62da60624c4fd18"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "e51e7948e9fe0f3ae30386f3c491758a"
server_key_iv: "e75cfae44953e665e44d3de4591d5b36"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.780936 12149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:24.798369 12149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:24.798704 12149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:24.798856 12149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:24.799069 12149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:24.799377 12149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:24.799436 12149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.799472 12149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:24.799502 12149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.805222 12149 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:43107
I20260810 21:10:24.805274 12363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:43107 every 8 connection(s)
I20260810 21:10:24.805568 12149 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:10:24.808977 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 12149
I20260810 21:10:24.809078 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:10:24.809267 28117 external_mini_cluster.cc:1428] Setting key cf345362c3d42510c929acd9eebb5fa0
I20260810 21:10:24.810195 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38289
--builtin_ntp_servers=127.27.117.84:41679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 21:10:24.810719 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.806208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:50731 (local address 127.27.117.126:38289)
0810 21:10:24.806266 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:24.806269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:24.806517 (+ 248us) server_negotiation.cc:407] Connection header received
0810 21:10:24.806949 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:24.806950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:24.806960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:24.807009 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:24.807592 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.808060 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.809100 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.809369 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.809443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.810392 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:24.810393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:24.810394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:24.810399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:24.810416 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:24.810421 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:24.810455 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:24.810577 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 21:10:24.810606 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 21:10:24.810814 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.806223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38289 (local address 127.27.117.65:50731)
0810 21:10:24.806396 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:24.806426 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:24.806869 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:24.807056 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:24.807059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:24.807067 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:24.807523 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.807526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.808139 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.808139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.808996 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.808998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.809446 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.809447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.809601 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.810215 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:10:24.810215 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:24.810242 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:10:24.810263 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:24.810490 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:24.810496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:24.810499 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:10:24.810554 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 21:10:24.810632 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260810 21:10:24.811296 12364 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38289
I20260810 21:10:24.811405 12364 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:24.811579 12364 heartbeater.cc:511] Master 127.27.117.126:38289 requested a full tablet report, sending...
I20260810 21:10:24.812135 12041 ts_manager.cc:194] Registered new tserver with Master: 448dec23cb0a4013b62da60624c4fd18 (127.27.117.65:43107)
I20260810 21:10:24.812764 12041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.65:50731
W20260810 21:10:24.892479 12371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:24.892666 12371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:24.892686 12371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:24.893901 12371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:24.893953 12371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:24.893978 12371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:10:24.895485 12371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:41679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:24.895723 12371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:24.896121 12371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:24.901700 12382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:24.901727 12380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:24.901732 12378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:24.901911 12371 server_base.cc:1034] running on GCE node
I20260810 21:10:24.902094 12371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:24.902311 12371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:24.903469 12371 hybrid_clock.cc:648] HybridClock initialized: now 1786396224903459 us; error 42 us; skew 500 ppm
I20260810 21:10:24.904618 12371 webserver.cc:460] Webserver started at http://127.27.117.66:34987/ using document root <none> and password file <none>
I20260810 21:10:24.904915 12371 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:24.904976 12371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:24.905126 12371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:24.906193 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "605a890002d94a2496ffd95492510f2d"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "40ad750398a3c6d6d87d2c69714e1ab0"
server_key_iv: "05f4f7f2d6f087582bc3fbc3f2674f9f"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.906560 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "605a890002d94a2496ffd95492510f2d"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "40ad750398a3c6d6d87d2c69714e1ab0"
server_key_iv: "05f4f7f2d6f087582bc3fbc3f2674f9f"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.907863 12371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:10:24.908568 12391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.908756 12371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:10:24.908887 12371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "605a890002d94a2496ffd95492510f2d"
format_stamp: "Formatted at 2026-08-10 21:10:24 on dist-test-slave-2kcd"
server_key: "40ad750398a3c6d6d87d2c69714e1ab0"
server_key_iv: "05f4f7f2d6f087582bc3fbc3f2674f9f"
server_key_version: "encryptionkey@0"
I20260810 21:10:24.908983 12371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:24.941035 12371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:24.941378 12371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:24.941540 12371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:24.941787 12371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:24.942147 12371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:24.942203 12371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.942239 12371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:24.942266 12371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:24.948243 12371 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:33533
I20260810 21:10:24.948302 12611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:33533 every 8 connection(s)
I20260810 21:10:24.948689 12371 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:10:24.953790 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.949484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:58293 (local address 127.27.117.126:38289)
0810 21:10:24.949530 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:24.949531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:24.949792 (+ 261us) server_negotiation.cc:407] Connection header received
0810 21:10:24.950280 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:24.950281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:24.950291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:24.950349 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:24.951044 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.951594 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.952538 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.952721 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.952772 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.953565 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:24.953566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:24.953567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:24.953571 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:24.953583 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:24.953587 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:24.953630 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:24.953712 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 21:10:24.953734 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:10:24.954030 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:24.949464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38289 (local address 127.27.117.66:58293)
0810 21:10:24.949658 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:24.949699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:24.950199 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:24.950399 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:24.950402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:24.950410 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:24.950960 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.950967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.951694 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.951694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.952463 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:24.952465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:24.952811 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:24.952811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:24.952966 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:24.953434 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:10:24.953434 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:24.953461 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:10:24.953482 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:24.953641 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:24.953646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:24.953648 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:24.953698 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 21:10:24.953775 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260810 21:10:24.954550 12612 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38289
I20260810 21:10:24.954653 12612 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:24.954849 12612 heartbeater.cc:511] Master 127.27.117.126:38289 requested a full tablet report, sending...
I20260810 21:10:24.955260 12041 ts_manager.cc:194] Registered new tserver with Master: 605a890002d94a2496ffd95492510f2d (127.27.117.66:33533)
I20260810 21:10:24.955654 12041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.66:58293
I20260810 21:10:24.956635 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 12371
I20260810 21:10:24.956724 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:10:24.956948 28117 external_mini_cluster.cc:1428] Setting key 6a875f29b289ecfcf25706435b64309a
I20260810 21:10:24.957811 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38289
--builtin_ntp_servers=127.27.117.84:41679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 21:10:25.038406 12618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:25.038596 12618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:25.038617 12618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:25.039748 12618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:25.039788 12618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:25.039812 12618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:10:25.041206 12618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:41679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:25.041441 12618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:25.041736 12618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:25.046536 12624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.046536 12630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.046553 12625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.046747 12618 server_base.cc:1034] running on GCE node
I20260810 21:10:25.046958 12618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:25.047223 12618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:25.048382 12618 hybrid_clock.cc:648] HybridClock initialized: now 1786396225048373 us; error 38 us; skew 500 ppm
I20260810 21:10:25.049631 12618 webserver.cc:460] Webserver started at http://127.27.117.67:35435/ using document root <none> and password file <none>
I20260810 21:10:25.049887 12618 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:25.049968 12618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:25.050100 12618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:25.051231 12618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "d5862d4b475d4f1f99267ac0394daee2"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "7697cc578d8f598ee555d18a68cedbb2"
server_key_iv: "cd6b212b9d9d31d24195d33c9efca3e3"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.051642 12618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "d5862d4b475d4f1f99267ac0394daee2"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "7697cc578d8f598ee555d18a68cedbb2"
server_key_iv: "cd6b212b9d9d31d24195d33c9efca3e3"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.052981 12618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:10:25.053670 12638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.053846 12618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:25.053947 12618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "d5862d4b475d4f1f99267ac0394daee2"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "7697cc578d8f598ee555d18a68cedbb2"
server_key_iv: "cd6b212b9d9d31d24195d33c9efca3e3"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.054028 12618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:25.070782 12618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:25.071061 12618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:25.071213 12618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:25.071432 12618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:25.071734 12618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:25.071789 12618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.071828 12618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:25.071854 12618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.077745 12618 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:46825
I20260810 21:10:25.077781 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:46825 every 8 connection(s)
I20260810 21:10:25.078102 12618 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:10:25.082940 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.078797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:59767 (local address 127.27.117.126:38289)
0810 21:10:25.078857 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.078859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.079103 (+ 244us) server_negotiation.cc:407] Connection header received
0810 21:10:25.079582 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.079583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.079592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.079645 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.080256 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:25.080741 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:25.081706 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:25.081892 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:25.081944 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:25.082614 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:25.082615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:25.082616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:25.082620 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:25.082645 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:25.082648 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:25.082690 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:25.082859 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 21:10:25.082885 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:10:25.083098 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.078767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38289 (local address 127.27.117.67:59767)
0810 21:10:25.078950 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:25.078989 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:25.079495 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:25.079699 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:25.079702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:25.079710 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:25.080180 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:25.080184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:25.080859 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:25.080860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:25.081629 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:25.081632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:25.081967 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:25.081968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:25.082070 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:25.082486 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:10:25.082486 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:25.082514 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:10:25.082534 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:25.082702 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:25.082706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:25.082723 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 21:10:25.082838 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 21:10:25.082922 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":99,"threads_started":1}
I20260810 21:10:25.083375 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 12618
I20260810 21:10:25.083469 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:10:25.083617 12825 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38289
I20260810 21:10:25.083683 28117 external_mini_cluster.cc:1428] Setting key 5cbde67da7a573a4cf7ffba042e4f198
I20260810 21:10:25.083714 12825 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:25.083956 12825 heartbeater.cc:511] Master 127.27.117.126:38289 requested a full tablet report, sending...
I20260810 21:10:25.084321 12041 ts_manager.cc:194] Registered new tserver with Master: d5862d4b475d4f1f99267ac0394daee2 (127.27.117.67:46825)
I20260810 21:10:25.084733 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:10:25.084877 12041 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.67:59767
W20260810 21:10:25.086563 12115 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:25.086076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.27.117.126:38289)
0810 21:10:25.086129 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.086131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.086247 (+ 116us) server_negotiation.cc:407] Connection header received
0810 21:10:25.086319 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.086320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.086330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.086378 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.086502 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58366: BlockingRecv error: recv got EOF from 127.0.0.1:58366 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:10:25.086615 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 12149
I20260810 21:10:25.112286 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 12371
I20260810 21:10:25.122849 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 12618
I20260810 21:10:25.130290 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 12000
2026-08-10T21:10:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (632 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T21:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:25Z Disabled control of system clock
I20260810 21:10:25.161396 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:44303
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:44303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 21:10:25.241134 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:25.241317 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:25.241338 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:25.242372 12840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:25.242401 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:25.242413 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:25.242425 12840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:25.242437 12840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:25.243633 12840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:44303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:44303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:25.243803 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:25.244014 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:25.249099 12854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.249063 12859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.249080 12856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.249258 12840 server_base.cc:1034] running on GCE node
I20260810 21:10:25.249634 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:25.249866 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:25.251036 12840 hybrid_clock.cc:648] HybridClock initialized: now 1786396225251031 us; error 39 us; skew 500 ppm
I20260810 21:10:25.252203 12840 webserver.cc:460] Webserver started at http://127.27.117.126:38841/ using document root <none> and password file <none>
I20260810 21:10:25.252437 12840 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:25.252518 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:25.252681 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:25.253750 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "6c785943ae294b53af4f6aae137e4f6d"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "ec774e254ed0b751513d7b6f7df1c9c5"
server_key_iv: "886cc8d5e9515a3d6ae595bf4840b859"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.254146 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "6c785943ae294b53af4f6aae137e4f6d"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "ec774e254ed0b751513d7b6f7df1c9c5"
server_key_iv: "886cc8d5e9515a3d6ae595bf4840b859"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.255362 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:10:25.256083 12868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.256254 12840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:10:25.256363 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "6c785943ae294b53af4f6aae137e4f6d"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "ec774e254ed0b751513d7b6f7df1c9c5"
server_key_iv: "886cc8d5e9515a3d6ae595bf4840b859"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.256439 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:25.260423 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:25.260679 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:25.260843 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:25.264894 12840 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:44303
I20260810 21:10:25.264953 12949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:44303 every 8 connection(s)
I20260810 21:10:25.265287 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:10:25.265931 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:25.268636 12951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d: Bootstrap starting.
I20260810 21:10:25.269261 12951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:25.269580 12951 log.cc:824] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:25.270252 12951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d: No bootstrap required, opened a new log
I20260810 21:10:25.271803 12951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c785943ae294b53af4f6aae137e4f6d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44303 } }
I20260810 21:10:25.271939 12951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:25.271963 12951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c785943ae294b53af4f6aae137e4f6d, State: Initialized, Role: FOLLOWER
I20260810 21:10:25.272060 12951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c785943ae294b53af4f6aae137e4f6d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44303 } }
I20260810 21:10:25.272140 12951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:10:25.272168 12951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:10:25.272212 12951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:25.272727 12951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c785943ae294b53af4f6aae137e4f6d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44303 } }
I20260810 21:10:25.272862 12951 leader_election.cc:302] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c785943ae294b53af4f6aae137e4f6d; no voters:
I20260810 21:10:25.273052 12951 leader_election.cc:288] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:10:25.273129 12955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:25.273303 12955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [term 1 LEADER]: Becoming Leader. State: Replica: 6c785943ae294b53af4f6aae137e4f6d, State: Running, Role: LEADER
I20260810 21:10:25.273303 12951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:10:25.273449 12955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c785943ae294b53af4f6aae137e4f6d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44303 } }
I20260810 21:10:25.274585 12958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c785943ae294b53af4f6aae137e4f6d. Latest consensus state: current_term: 1 leader_uuid: "6c785943ae294b53af4f6aae137e4f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c785943ae294b53af4f6aae137e4f6d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44303 } } }
I20260810 21:10:25.274681 12958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:25.274928 12974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:25.275069 12957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c785943ae294b53af4f6aae137e4f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c785943ae294b53af4f6aae137e4f6d" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 44303 } } }
I20260810 21:10:25.275156 12957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:25.275435 12974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:10:25.275434 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 12840
I20260810 21:10:25.275549 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:10:25.275736 28117 external_mini_cluster.cc:1428] Setting key c65d640f64fa9d7b7b17514557dbe3ef
I20260810 21:10:25.277045 12974 catalog_manager.cc:1347] Generated new cluster ID: 4c9f68a0ee644941afadeed3b2a5fd69
I20260810 21:10:25.277101 12974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:10:25.278523 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.276181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.27.117.126:44303)
0810 21:10:25.276353 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.276361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.277072 (+ 711us) server_negotiation.cc:407] Connection header received
0810 21:10:25.277212 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.277220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.277261 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.277388 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.278096 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:25.278099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:25.278100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:25.278103 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:25.278143 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:25.278152 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:25.278219 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:25.278292 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 21:10:25.278351 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260810 21:10:25.282275 12974 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:10:25.282413 12974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:25.286999 12974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c785943ae294b53af4f6aae137e4f6d: Generated new TSK 0
I20260810 21:10:25.287161 12974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:10:25.289250 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44303
--builtin_ntp_servers=127.27.117.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 21:10:25.378041 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:25.378230 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:25.378252 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:25.379400 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:25.379451 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:25.379475 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:10:25.380860 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:25.381074 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:25.381374 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:25.387410 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.387403 12996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.387624 12989 server_base.cc:1034] running on GCE node
W20260810 21:10:25.387473 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.387904 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:25.388209 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:25.389379 12989 hybrid_clock.cc:648] HybridClock initialized: now 1786396225389363 us; error 27 us; skew 500 ppm
I20260810 21:10:25.390684 12989 webserver.cc:460] Webserver started at http://127.27.117.65:37275/ using document root <none> and password file <none>
I20260810 21:10:25.390918 12989 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:25.390985 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:25.391152 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:25.392194 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "742338e0b9934656b2d1b12bfd21617d"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "d578cfd4d4899dd3d4fd3b50563bcca5"
server_key_iv: "8f7835ef98bdfef321bdb806f72fb79f"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.392565 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "742338e0b9934656b2d1b12bfd21617d"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "d578cfd4d4899dd3d4fd3b50563bcca5"
server_key_iv: "8f7835ef98bdfef321bdb806f72fb79f"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.393962 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:10:25.394742 13017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.395015 12989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:10:25.395113 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "742338e0b9934656b2d1b12bfd21617d"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "d578cfd4d4899dd3d4fd3b50563bcca5"
server_key_iv: "8f7835ef98bdfef321bdb806f72fb79f"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.395179 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:25.399646 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:25.399920 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:25.400044 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:25.400238 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:25.400545 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:25.400597 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.400624 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:25.400642 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.407842 12989 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:41951
I20260810 21:10:25.407938 13211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:41951 every 8 connection(s)
I20260810 21:10:25.408284 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:10:25.411363 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.409148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:39905 (local address 127.27.117.126:44303)
0810 21:10:25.409219 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.409221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.409467 (+ 246us) server_negotiation.cc:407] Connection header received
0810 21:10:25.409902 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.409904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.409911 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.409997 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.411123 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:25.411124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:25.411124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:25.411128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:25.411145 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:25.411150 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:25.411217 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:25.411275 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 21:10:25.411290 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:10:25.411505 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.409136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:44303 (local address 127.27.117.65:39905)
0810 21:10:25.409331 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:25.409368 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:25.409801 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:25.410042 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:25.410045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:25.410052 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:25.410985 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:10:25.410985 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:25.411012 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:10:25.411038 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:25.411214 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:25.411216 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:25.411218 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:25.411267 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 21:10:25.411312 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260810 21:10:25.412014 13212 heartbeater.cc:347] Connected to a master server at 127.27.117.126:44303
I20260810 21:10:25.412106 13212 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:25.412249 13212 heartbeater.cc:511] Master 127.27.117.126:44303 requested a full tablet report, sending...
I20260810 21:10:25.412746 12895 ts_manager.cc:194] Registered new tserver with Master: 742338e0b9934656b2d1b12bfd21617d (127.27.117.65:41951)
I20260810 21:10:25.415330 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 12989
I20260810 21:10:25.415436 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:10:25.415611 28117 external_mini_cluster.cc:1428] Setting key ff52e5fefea3b7f9fed7117a7c11e68f
I20260810 21:10:25.416494 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44303
--builtin_ntp_servers=127.27.117.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 21:10:25.498594 13220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:25.498899 13220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:25.498939 13220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:25.500064 13220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:25.500102 13220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:25.500128 13220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:10:25.501509 13220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:25.501677 13220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:25.501895 13220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:25.506907 13228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.507048 13232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.506942 13229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.507208 13220 server_base.cc:1034] running on GCE node
I20260810 21:10:25.507436 13220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:25.507682 13220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:25.508853 13220 hybrid_clock.cc:648] HybridClock initialized: now 1786396225508848 us; error 37 us; skew 500 ppm
I20260810 21:10:25.510084 13220 webserver.cc:460] Webserver started at http://127.27.117.66:42767/ using document root <none> and password file <none>
I20260810 21:10:25.510344 13220 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:25.510414 13220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:25.510576 13220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:25.511649 13220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "5b86514a4de14c249fdadca2305d42aa"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "a02da0b5193970b9d5d557394faf6b55"
server_key_iv: "c0cf8928ef551d2b4e104d127caf1c0b"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.511991 13220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "5b86514a4de14c249fdadca2305d42aa"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "a02da0b5193970b9d5d557394faf6b55"
server_key_iv: "c0cf8928ef551d2b4e104d127caf1c0b"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.513378 13220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:10:25.514128 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.514307 13220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:25.514397 13220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "5b86514a4de14c249fdadca2305d42aa"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "a02da0b5193970b9d5d557394faf6b55"
server_key_iv: "c0cf8928ef551d2b4e104d127caf1c0b"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.514477 13220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:25.517803 13220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:25.518049 13220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:25.518183 13220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:25.518424 13220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:25.518709 13220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:25.518764 13220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.518803 13220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:25.518829 13220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.524832 13220 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:38833
I20260810 21:10:25.524868 13435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:38833 every 8 connection(s)
I20260810 21:10:25.525156 13220 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:10:25.527802 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.525868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:47901 (local address 127.27.117.126:44303)
0810 21:10:25.525915 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.525917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.526110 (+ 193us) server_negotiation.cc:407] Connection header received
0810 21:10:25.526512 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.526512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.526518 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.526582 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.527599 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:25.527601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:25.527601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:25.527605 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:25.527621 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:25.527626 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:25.527684 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:25.527735 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 21:10:25.527748 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:10:25.527951 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.525820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:44303 (local address 127.27.117.66:47901)
0810 21:10:25.526001 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:25.526052 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:25.526430 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:25.526647 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:25.526650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:25.526657 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:25.527461 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:10:25.527461 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:25.527486 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:10:25.527511 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:25.527683 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:25.527685 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:25.527687 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:25.527732 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 21:10:25.527775 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260810 21:10:25.528458 13437 heartbeater.cc:347] Connected to a master server at 127.27.117.126:44303
I20260810 21:10:25.528538 13437 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:25.528728 13437 heartbeater.cc:511] Master 127.27.117.126:44303 requested a full tablet report, sending...
I20260810 21:10:25.529141 12895 ts_manager.cc:194] Registered new tserver with Master: 5b86514a4de14c249fdadca2305d42aa (127.27.117.66:38833)
I20260810 21:10:25.531857 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 13220
I20260810 21:10:25.531961 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:10:25.532138 28117 external_mini_cluster.cc:1428] Setting key 8a078a9f33135a93ffff7d136585417f
I20260810 21:10:25.533039 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44303
--builtin_ntp_servers=127.27.117.84:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 21:10:25.613955 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:25.614176 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:25.614215 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:25.615368 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:25.615437 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:25.615483 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:10:25.616892 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:25.617148 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:25.617430 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:25.622447 13450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.622532 13440 server_base.cc:1034] running on GCE node
W20260810 21:10:25.622444 13452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:25.622442 13449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:25.622831 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:25.623054 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:25.624187 13440 hybrid_clock.cc:648] HybridClock initialized: now 1786396225624180 us; error 31 us; skew 500 ppm
I20260810 21:10:25.625422 13440 webserver.cc:460] Webserver started at http://127.27.117.67:42275/ using document root <none> and password file <none>
I20260810 21:10:25.625648 13440 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:25.625692 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:25.625883 13440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:25.626879 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "864813ecf2834a71adce91c3fc5c6d80"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "c775724c5861c6f848e89b4905c2b9c5"
server_key_iv: "9991e378be263017575858166b833dcb"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.627223 13440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "864813ecf2834a71adce91c3fc5c6d80"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "c775724c5861c6f848e89b4905c2b9c5"
server_key_iv: "9991e378be263017575858166b833dcb"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.628458 13440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:10:25.629268 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.629447 13440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.629534 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "864813ecf2834a71adce91c3fc5c6d80"
format_stamp: "Formatted at 2026-08-10 21:10:25 on dist-test-slave-2kcd"
server_key: "c775724c5861c6f848e89b4905c2b9c5"
server_key_iv: "9991e378be263017575858166b833dcb"
server_key_version: "encryptionkey@0"
I20260810 21:10:25.629603 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:25.632902 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:25.633138 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:25.633266 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:25.633476 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:25.633759 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:25.633790 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.633810 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:25.633822 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:25.639583 13440 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:39487
I20260810 21:10:25.639619 13665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:39487 every 8 connection(s)
I20260810 21:10:25.639923 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:10:25.642669 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.640649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:34227 (local address 127.27.117.126:44303)
0810 21:10:25.640713 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.640714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.640922 (+ 208us) server_negotiation.cc:407] Connection header received
0810 21:10:25.641335 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.641336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.641342 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.641399 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.642508 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:25.642509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:25.642509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:10:25.642512 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:25.642521 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:10:25.642525 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:25.642566 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:25.642609 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 21:10:25.642619 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:10:25.642927 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:25.640609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:44303 (local address 127.27.117.67:34227)
0810 21:10:25.640807 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:25.640852 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:25.641235 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:25.641442 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:25.641445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:25.641453 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:25.642383 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:10:25.642384 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:25.642408 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:10:25.642433 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:25.642565 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:25.642567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:25.642569 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:25.642617 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 21:10:25.642657 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260810 21:10:25.643432 13666 heartbeater.cc:347] Connected to a master server at 127.27.117.126:44303
I20260810 21:10:25.643530 13666 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:25.643695 13666 heartbeater.cc:511] Master 127.27.117.126:44303 requested a full tablet report, sending...
I20260810 21:10:25.644030 12895 ts_manager.cc:194] Registered new tserver with Master: 864813ecf2834a71adce91c3fc5c6d80 (127.27.117.67:39487)
I20260810 21:10:25.648048 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 13440
I20260810 21:10:25.648125 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:10:25.648319 28117 external_mini_cluster.cc:1428] Setting key ed5f5866724becd262c2b1632fe893ef
I20260810 21:10:25.649284 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 21:10:25.651036 12987 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:10:25.650598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.27.117.126:44303)
0810 21:10:25.650666 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:25.650668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:25.650754 (+ 86us) server_negotiation.cc:407] Connection header received
0810 21:10:25.650794 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:25.650795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:25.650798 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:25.650845 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:25.650975 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44134: BlockingRecv error: recv got EOF from 127.0.0.1:44134 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:10:25.651142 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 12989
I20260810 21:10:25.659519 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 13220
I20260810 21:10:25.677814 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 13440
I20260810 21:10:25.686457 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 12840
2026-08-10T21:10:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (560 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:10:25 dist-test-slave-2kcd krb5kdc[13686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:10:25 dist-test-slave-2kcd krb5kdc[13686](info): set up 2 sockets
Aug 10 21:10:25 dist-test-slave-2kcd krb5kdc[13686](info): commencing operation
krb5kdc: starting...
W20260810 21:10:26.961977 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:10:27.083249 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35711
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:27.162796 13707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:27.163014 13707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:27.163050 13707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:27.164162 13707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:10:27.164245 13707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:27.164270 13707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:27.164299 13707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:10:27.164330 13707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:10:27.165800 13707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:27.166082 13707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:27.166309 13707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:27.171128 13721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:27.171118 13715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.171200 13707 server_base.cc:1034] running on GCE node
W20260810 21:10:27.171116 13716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.171542 13707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:27.171829 13707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:27.172991 13707 hybrid_clock.cc:648] HybridClock initialized: now 1786396227172974 us; error 45 us; skew 500 ppm
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:27.175396 13707 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:10:27.175835 13707 webserver.cc:460] Webserver started at http://127.27.117.126:34531/ using document root <none> and password file <none>
I20260810 21:10:27.176025 13707 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:27.176066 13707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:27.176165 13707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:27.177194 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "3e84660d16274f27baefccd4df01ea99"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "cb43aee24f4ccdbefc1e629878c15e3e"
server_key_iv: "c71e775a70961d87940837d13ce501c3"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.177484 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "3e84660d16274f27baefccd4df01ea99"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "cb43aee24f4ccdbefc1e629878c15e3e"
server_key_iv: "c71e775a70961d87940837d13ce501c3"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.178777 13707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:10:27.179454 13733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.179639 13707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:27.179713 13707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "3e84660d16274f27baefccd4df01ea99"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "cb43aee24f4ccdbefc1e629878c15e3e"
server_key_iv: "c71e775a70961d87940837d13ce501c3"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.179764 13707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:27.196038 13707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:27.197715 13707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:27.197825 13707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:27.201989 13707 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35711
I20260810 21:10:27.202031 13826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35711 every 8 connection(s)
I20260810 21:10:27.202344 13707 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:10:27.203240 13828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:27.205467 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99: Bootstrap starting.
I20260810 21:10:27.206005 13828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:27.206244 13828 log.cc:824] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:27.206893 13828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99: No bootstrap required, opened a new log
I20260810 21:10:27.208108 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 13707
I20260810 21:10:27.208181 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:10:27.208359 28117 external_mini_cluster.cc:1428] Setting key e16984c86566e794d63448b252eb7414
I20260810 21:10:27.208565 13828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e84660d16274f27baefccd4df01ea99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:10:27.208720 13828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:27.208757 13828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e84660d16274f27baefccd4df01ea99, State: Initialized, Role: FOLLOWER
I20260810 21:10:27.208906 13828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e84660d16274f27baefccd4df01ea99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:10:27.209004 13828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:10:27.209059 13828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:10:27.209100 13828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:27.209923 13828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e84660d16274f27baefccd4df01ea99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:10:27.210008 13828 leader_election.cc:302] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e84660d16274f27baefccd4df01ea99; no voters:
I20260810 21:10:27.210191 13828 leader_election.cc:288] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:10:27.210258 13836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:27.210417 13836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [term 1 LEADER]: Becoming Leader. State: Replica: 3e84660d16274f27baefccd4df01ea99, State: Running, Role: LEADER
I20260810 21:10:27.210436 13828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:10:27.210507 13836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e84660d16274f27baefccd4df01ea99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:10:27.210937 13837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e84660d16274f27baefccd4df01ea99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e84660d16274f27baefccd4df01ea99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } } }
I20260810 21:10:27.211014 13837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:27.211465 13838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e84660d16274f27baefccd4df01ea99. Latest consensus state: current_term: 1 leader_uuid: "3e84660d16274f27baefccd4df01ea99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e84660d16274f27baefccd4df01ea99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } } }
I20260810 21:10:27.211536 13838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99 [sys.catalog]: This master's current role is: LEADER
I20260810 21:10:27.211670 13849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:10:27.212189 13849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:10:27.214174 13849 catalog_manager.cc:1347] Generated new cluster ID: 840fe13a072e4c788f856b08b7b47d4f
I20260810 21:10:27.214237 13849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:27.219966 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.208877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.27.117.126:35711)
0810 21:10:27.209070 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.209080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.209189 (+ 109us) server_negotiation.cc:407] Connection header received
0810 21:10:27.209343 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.209349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.209398 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.209528 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:27.210198 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.211801 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.212896 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.213471 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.213617 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.216398 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:27.216402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:27.216402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:27.216414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:27.218372 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.218775 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.218777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.218778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.218889 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.219115 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.219115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.219116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.219386 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:27.219533 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.219659 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.219782 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260810 21:10:27.243046 13849 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:10:27.243584 13849 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:10:27.253216 13849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99: Generated new TSK 0
I20260810 21:10:27.253386 13849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:10:27.280961 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35711
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:27.363603 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:27.363790 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:27.363812 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:27.365011 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:27.365054 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:27.365082 13872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:10:27.366595 13872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:27.366840 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:27.367069 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:27.372136 13879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:27.372129 13881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.372332 13872 server_base.cc:1034] running on GCE node
W20260810 21:10:27.372174 13889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.372571 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:27.372888 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:27.374039 13872 hybrid_clock.cc:648] HybridClock initialized: now 1786396227374017 us; error 33 us; skew 500 ppm
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:27.376619 13872 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:10:27.377117 13872 webserver.cc:460] Webserver started at http://127.27.117.65:34419/ using document root <none> and password file <none>
I20260810 21:10:27.377368 13872 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:27.377414 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:27.377585 13872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:27.378599 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "aee6dfb6eb2aca95aa7e22cc8662d483"
server_key_iv: "203117c9051132d6c4936f692423a8e0"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.378963 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "aee6dfb6eb2aca95aa7e22cc8662d483"
server_key_iv: "203117c9051132d6c4936f692423a8e0"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.380290 13872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:10:27.381131 13902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.381364 13872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:10:27.381465 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "aee6dfb6eb2aca95aa7e22cc8662d483"
server_key_iv: "203117c9051132d6c4936f692423a8e0"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.381526 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:27.398615 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:27.401149 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:27.401273 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:27.401463 13872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:27.401808 13872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:27.401850 13872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.401898 13872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:27.401928 13872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.408351 13872 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:44839
I20260810 21:10:27.408416 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:44839 every 8 connection(s)
I20260810 21:10:27.408793 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:27.416252 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 13872
I20260810 21:10:27.416354 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:10:27.416577 28117 external_mini_cluster.cc:1428] Setting key 84ccf59cc100e0bf805408e6ac48fea9
I20260810 21:10:27.418915 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.409557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35711 (local address 127.27.117.65:38729)
0810 21:10:27.409738 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.409819 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.410367 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.410628 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.410631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.411330 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:27.411562 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.411567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.412394 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.412394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.413255 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.413257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.413624 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.413625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.413729 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.414201 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:27.414209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:27.415581 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:27.417587 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:27.417594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:27.417596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:27.417826 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:27.418041 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:27.418041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:27.418042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:27.418133 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:27.418419 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:27.418425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:27.418539 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.418628 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.418751 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260810 21:10:27.418919 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.409600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:38729 (local address 127.27.117.126:35711)
0810 21:10:27.409695 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.409698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.409894 (+ 196us) server_negotiation.cc:407] Connection header received
0810 21:10:27.410477 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.410478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.410490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.410555 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:27.411666 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.412283 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.413340 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.413545 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.413608 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.415674 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:27.415676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:27.415676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:27.415684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:27.417503 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.417909 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.417910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.417911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.417976 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.418197 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.418197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.418198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.418350 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:27.418417 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.418672 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.418770 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 21:10:27.419565 14107 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35711
I20260810 21:10:27.419657 14107 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:27.419855 14107 heartbeater.cc:511] Master 127.27.117.126:35711 requested a full tablet report, sending...
I20260810 21:10:27.420380 13760 ts_manager.cc:194] Registered new tserver with Master: b8d874cb6a6c4dbab8d98c3a78bab98d (127.27.117.65:44839)
I20260810 21:10:27.421038 13760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:38729
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:10:27.443764 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35711
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:27.524044 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:27.524219 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:27.524241 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:27.525398 14113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:27.525436 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:27.525462 14113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:10:27.526852 14113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:27.527060 14113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:27.527261 14113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:27.532186 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:27.532250 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:27.532209 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.532431 14113 server_base.cc:1034] running on GCE node
I20260810 21:10:27.532627 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:27.532912 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:27.534051 14113 hybrid_clock.cc:648] HybridClock initialized: now 1786396227534039 us; error 30 us; skew 500 ppm
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:27.536667 14113 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:10:27.537161 14113 webserver.cc:460] Webserver started at http://127.27.117.66:42979/ using document root <none> and password file <none>
I20260810 21:10:27.537384 14113 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:27.537431 14113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:27.537572 14113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:27.538553 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "063bc4065d254c488c13947879ff12a4"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "8f34714d3094f9701d7579bfc3cd0993"
server_key_iv: "65e15541fb3ce3fdc93baeb596350041"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.538925 14113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "063bc4065d254c488c13947879ff12a4"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "8f34714d3094f9701d7579bfc3cd0993"
server_key_iv: "65e15541fb3ce3fdc93baeb596350041"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.540197 14113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:10:27.540946 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.541102 14113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:10:27.541184 14113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "063bc4065d254c488c13947879ff12a4"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "8f34714d3094f9701d7579bfc3cd0993"
server_key_iv: "65e15541fb3ce3fdc93baeb596350041"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.541235 14113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:27.560153 14113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:27.561872 14113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:27.561978 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:27.562150 14113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:27.562450 14113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:27.562484 14113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.562505 14113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:27.562561 14113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.568079 14113 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:42857
I20260810 21:10:27.568114 14327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:42857 every 8 connection(s)
I20260810 21:10:27.568403 14113 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:10:27.568784 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 14113
I20260810 21:10:27.568908 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:10:27.569150 28117 external_mini_cluster.cc:1428] Setting key a51e5b671abed35a375f5395e9e723b9
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
I20260810 21:10:27.578742 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.569301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:56395 (local address 127.27.117.126:35711)
0810 21:10:27.569362 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.569364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.569482 (+ 118us) server_negotiation.cc:407] Connection header received
0810 21:10:27.570029 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.570030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.570046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.570091 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:27.571339 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.571943 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.572944 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.573153 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.573240 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.575784 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:27.575785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:27.575786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:27.575792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:27.577263 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.577754 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.577755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.577755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.577807 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.578101 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.578102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.578102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.578203 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:27.578254 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.578591 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.578681 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:10:27.578975 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.569168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35711 (local address 127.27.117.66:56395)
0810 21:10:27.569345 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.569390 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.569924 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.570158 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.570162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.570994 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:27.571252 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.571258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.572038 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.572039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.572840 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.572842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.573251 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.573252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.573380 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.573850 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:27.573857 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:27.575688 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:27.577351 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:27.577359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:27.577361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:27.577672 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:27.577880 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:27.577881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:27.577882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:27.578026 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:27.578271 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:27.578277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:27.578426 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.578552 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.578722 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260810 21:10:27.579466 14329 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35711
I20260810 21:10:27.579568 14329 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:27.579780 14329 heartbeater.cc:511] Master 127.27.117.126:35711 requested a full tablet report, sending...
I20260810 21:10:27.580163 13760 ts_manager.cc:194] Registered new tserver with Master: 063bc4065d254c488c13947879ff12a4 (127.27.117.66:42857)
I20260810 21:10:27.580540 13760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:56395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:10:27.595126 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35711
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:10:27.675345 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:10:27.675540 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:10:27.675562 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:10:27.676585 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:10:27.676617 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:10:27.676641 14336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:10:27.678025 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:10:27.678213 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:10:27.678375 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:10:27.683311 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.683425 14336 server_base.cc:1034] running on GCE node
W20260810 21:10:27.683321 14346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:10:27.683378 14354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:10:27.683750 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:10:27.683998 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:10:27.685132 14336 hybrid_clock.cc:648] HybridClock initialized: now 1786396227685119 us; error 28 us; skew 500 ppm
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:10:27.687683 14336 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:10:27.688138 14336 webserver.cc:460] Webserver started at http://127.27.117.67:39433/ using document root <none> and password file <none>
I20260810 21:10:27.688367 14336 fs_manager.cc:359] Metadata directory not provided
I20260810 21:10:27.688416 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:10:27.688593 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:10:27.689747 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "bc02fc41f5074b338c54e94977f165a9"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "7d4f7db376b02a9ac4a84e53460cc289"
server_key_iv: "2ba97bfbf1e7a7d993b0573435048999"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.690112 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "bc02fc41f5074b338c54e94977f165a9"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "7d4f7db376b02a9ac4a84e53460cc289"
server_key_iv: "2ba97bfbf1e7a7d993b0573435048999"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.691397 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:10:27.692094 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.692291 14336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:10:27.692380 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "bc02fc41f5074b338c54e94977f165a9"
format_stamp: "Formatted at 2026-08-10 21:10:27 on dist-test-slave-2kcd"
server_key: "7d4f7db376b02a9ac4a84e53460cc289"
server_key_iv: "2ba97bfbf1e7a7d993b0573435048999"
server_key_version: "encryptionkey@0"
I20260810 21:10:27.692449 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:10:27.710577 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:10:27.712937 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:10:27.713093 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:10:27.713315 14336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:10:27.713650 14336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:10:27.713701 14336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.713737 14336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:10:27.713762 14336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:10:27.719599 14336 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:41361
I20260810 21:10:27.719635 14570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:41361 every 8 connection(s)
I20260810 21:10:27.720005 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:10:27 dist-test-slave-2kcd krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396227, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:10:27.728533 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.720764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:33405 (local address 127.27.117.126:35711)
0810 21:10:27.720847 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.720850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.721061 (+ 211us) server_negotiation.cc:407] Connection header received
0810 21:10:27.721503 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.721504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.721515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.721567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:27.722392 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.722869 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.723830 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.724017 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.724066 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.726052 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:27.726054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:27.726054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:27.726061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:27.727446 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.727775 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.727776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.727776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.727824 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.728031 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.728032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.728032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.728127 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:27.728176 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.728388 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.728449 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 21:10:27.728701 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.720755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35711 (local address 127.27.117.67:33405)
0810 21:10:27.720941 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.720975 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.721418 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.721614 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.721617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.722148 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:10:27.722319 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.722324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.722947 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.722948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.723758 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.723760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.724088 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.724089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.724188 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.724605 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:10:27.724613 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:10:27.725967 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:10:27.727514 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:27.727519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:27.727520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:27.727713 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:27.727881 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:10:27.727882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:10:27.727882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:10:27.727978 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:10:27.728182 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:10:27.728186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:10:27.728283 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.728361 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.728467 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260810 21:10:27.729224 14571 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35711
I20260810 21:10:27.729305 14571 heartbeater.cc:464] Registering TS with master...
I20260810 21:10:27.729482 14571 heartbeater.cc:511] Master 127.27.117.126:35711 requested a full tablet report, sending...
I20260810 21:10:27.729748 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 14336
I20260810 21:10:27.729840 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:10:27.729885 13760 ts_manager.cc:194] Registered new tserver with Master: bc02fc41f5074b338c54e94977f165a9 (127.27.117.67:41361)
I20260810 21:10:27.730026 28117 external_mini_cluster.cc:1428] Setting key 576557995c9a00b0ee8264796c26e8a3
I20260810 21:10:27.730294 13760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:33405
I20260810 21:10:27.730957 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:10:27.738364 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.732183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.27.117.126:35711)
0810 21:10:27.732216 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.732217 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.732361 (+ 144us) server_negotiation.cc:407] Connection header received
0810 21:10:27.732459 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.732460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.732470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.732539 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:10:27.733154 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.733595 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.734294 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.734484 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.734567 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.735689 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:10:27.735690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:10:27.735690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:10:27.735697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:10:27.737211 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.737576 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.737577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.737578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.737637 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:10:27.737841 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:10:27.737842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:10:27.737842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:10:27.737987 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:10:27.738092 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.738203 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.738274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:10:27.740032 13759 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:10:27.741747 13759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:10:27.747874 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.744063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:41361 (local address 127.0.0.1:60968)
0810 21:10:27.744386 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.744394 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.744453 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.744605 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.744606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.744608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:27.744661 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.744663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.745645 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.745646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.746645 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.746647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.747589 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.747590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.747667 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.747679 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.747746 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.747795 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":33,"threads_started":1}
I20260810 21:10:27.747872 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.744217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.27.117.67:41361)
0810 21:10:27.744426 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.744430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.744440 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:10:27.744518 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.744519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.744533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.744604 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:27.744795 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.745538 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.746798 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.747510 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.747589 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.747641 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.747765 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.747801 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260810 21:10:27.748026 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.743991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.27.117.66:42857)
0810 21:10:27.744210 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.744214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.744225 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:10:27.744353 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.744355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.744373 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.744459 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:27.744655 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.745858 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.746972 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.747482 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.747557 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.747611 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.747908 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.747948 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
I20260810 21:10:27.748052 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.743871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:42857 (local address 127.0.0.1:44374)
0810 21:10:27.744144 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.744155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.744220 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.744479 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.744482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.744485 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:27.744554 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.744557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.745972 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.745973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.746883 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.746886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.747646 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.747647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.747721 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.747725 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.747771 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.747813 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":52,"threads_started":1}
I20260810 21:10:27.748438 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.743863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:44839 (local address 127.0.0.1:45116)
0810 21:10:27.744010 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.744047 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.744109 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.744920 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.744921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.744923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:27.744969 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.744970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.746296 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.746296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.747297 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.747300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.748182 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.748183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.748247 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.748251 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.748337 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.748376 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":9,"thread_start_us":54,"threads_started":1}
I20260810 21:10:27.748824 14469 tablet_service.cc:1467] Processing CreateTablet for tablet 0e74cac343d74cebb15a91b8e9c4689b (DEFAULT_TABLE table=test-table [id=5cf94021c7ac456b8fbd2485b685dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:27.749200 14469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e74cac343d74cebb15a91b8e9c4689b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:27.749467 14219 tablet_service.cc:1467] Processing CreateTablet for tablet 0e74cac343d74cebb15a91b8e9c4689b (DEFAULT_TABLE table=test-table [id=5cf94021c7ac456b8fbd2485b685dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:27.749771 14219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e74cac343d74cebb15a91b8e9c4689b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:27.751607 14591 tablet_bootstrap.cc:492] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Bootstrap starting.
I20260810 21:10:27.751698 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.743970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.27.117.65:44839)
0810 21:10:27.744592 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.744717 (+ 125us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.744729 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:10:27.744774 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.744799 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.744816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.745070 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:27.745146 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.746196 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.747386 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.748080 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.748141 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.748213 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.748355 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.748391 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":81,"threads_started":1}
I20260810 21:10:27.752313 14591 tablet_bootstrap.cc:654] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:27.752307 13988 tablet_service.cc:1467] Processing CreateTablet for tablet 0e74cac343d74cebb15a91b8e9c4689b (DEFAULT_TABLE table=test-table [id=5cf94021c7ac456b8fbd2485b685dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:10:27.752375 14592 tablet_bootstrap.cc:492] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Bootstrap starting.
I20260810 21:10:27.752655 13988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e74cac343d74cebb15a91b8e9c4689b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:10:27.752930 14591 log.cc:824] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:27.753062 14592 tablet_bootstrap.cc:654] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:27.753508 14592 log.cc:824] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:27.754206 14592 tablet_bootstrap.cc:492] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: No bootstrap required, opened a new log
I20260810 21:10:27.754289 14592 ts_tablet_manager.cc:1397] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:10:27.755227 14595 tablet_bootstrap.cc:492] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: Bootstrap starting.
I20260810 21:10:27.755846 14591 tablet_bootstrap.cc:492] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: No bootstrap required, opened a new log
I20260810 21:10:27.755901 14591 ts_tablet_manager.cc:1397] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 21:10:27.755829 14592 raft_consensus.cc:348] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.756016 14592 raft_consensus.cc:374] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:27.756064 14592 raft_consensus.cc:729] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063bc4065d254c488c13947879ff12a4, State: Initialized, Role: FOLLOWER
I20260810 21:10:27.756181 14592 consensus_queue.cc:260] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.756479 14592 ts_tablet_manager.cc:1428] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:10:27.756502 14329 heartbeater.cc:503] Master 127.27.117.126:35711 was elected leader, sending a full tablet report...
I20260810 21:10:27.757157 14595 tablet_bootstrap.cc:654] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: Neither blocks nor log segments found. Creating new log.
I20260810 21:10:27.757705 14595 log.cc:824] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:10:27.758029 14591 raft_consensus.cc:348] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.758170 14591 raft_consensus.cc:374] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:27.758217 14591 raft_consensus.cc:729] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc02fc41f5074b338c54e94977f165a9, State: Initialized, Role: FOLLOWER
I20260810 21:10:27.758327 14591 consensus_queue.cc:260] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.758477 14595 tablet_bootstrap.cc:492] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: No bootstrap required, opened a new log
I20260810 21:10:27.758548 14595 ts_tablet_manager.cc:1397] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 21:10:27.758641 14571 heartbeater.cc:503] Master 127.27.117.126:35711 was elected leader, sending a full tablet report...
I20260810 21:10:27.758626 14591 ts_tablet_manager.cc:1428] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:10:27.760447 14595 raft_consensus.cc:348] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.760574 14595 raft_consensus.cc:374] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:10:27.760596 14595 raft_consensus.cc:729] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d874cb6a6c4dbab8d98c3a78bab98d, State: Initialized, Role: FOLLOWER
I20260810 21:10:27.760664 14595 consensus_queue.cc:260] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.760942 14595 ts_tablet_manager.cc:1428] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:10:27.760947 14107 heartbeater.cc:503] Master 127.27.117.126:35711 was elected leader, sending a full tablet report...
W20260810 21:10:27.819083 14331 tablet.cc:2369] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:10:27.909591 14108 tablet.cc:2369] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:10:27.953078 14602 raft_consensus.cc:482] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:10:27.953190 14602 raft_consensus.cc:504] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.953637 14602 leader_election.cc:288] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 063bc4065d254c488c13947879ff12a4 (127.27.117.66:42857), b8d874cb6a6c4dbab8d98c3a78bab98d (127.27.117.65:44839)
I20260810 21:10:27.956527 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.953993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:55415 (local address 127.27.117.65:44839)
0810 21:10:27.954057 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.954060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.954187 (+ 127us) server_negotiation.cc:407] Connection header received
0810 21:10:27.954221 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.954222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.954232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.954289 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:27.954449 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.954925 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.955828 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.956175 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.956231 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.956261 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.956417 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.956461 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:10:27.956534 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.953877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:44839 (local address 127.27.117.67:55415)
0810 21:10:27.954114 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.954124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.954183 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.954314 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.954315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.954316 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:27.954382 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.954383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.955007 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.955008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.955717 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.955718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.956267 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.956268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.956345 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.956348 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.956392 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.956422 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":135,"threads_started":1}
I20260810 21:10:27.956606 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.953911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:45225 (local address 127.27.117.66:42857)
0810 21:10:27.953963 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.953965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.953978 (+ 13us) server_negotiation.cc:407] Connection header received
0810 21:10:27.954020 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.954021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.954030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.954097 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:10:27.954316 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.954850 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.955804 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.956180 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.956273 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.956281 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.956478 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.956521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:10:27.956622 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.953835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:42857 (local address 127.27.117.67:45225)
0810 21:10:27.953898 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.953919 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 21:10:27.953968 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:10:27.954111 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:10:27.954111 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:10:27.954113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:10:27.954212 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.954213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.954928 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.954929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.955705 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:10:27.955707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.956285 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.956286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:10:27.956364 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.956366 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:10:27.956451 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 21:10:27.956487 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 21:10:27.956969 14020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e74cac343d74cebb15a91b8e9c4689b" candidate_uuid: "bc02fc41f5074b338c54e94977f165a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" is_pre_election: true
I20260810 21:10:27.956971 14248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e74cac343d74cebb15a91b8e9c4689b" candidate_uuid: "bc02fc41f5074b338c54e94977f165a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063bc4065d254c488c13947879ff12a4" is_pre_election: true
I20260810 21:10:27.957123 14020 raft_consensus.cc:2454] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc02fc41f5074b338c54e94977f165a9 in term 0.
I20260810 21:10:27.957123 14248 raft_consensus.cc:2454] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc02fc41f5074b338c54e94977f165a9 in term 0.
I20260810 21:10:27.957391 14368 leader_election.cc:302] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d874cb6a6c4dbab8d98c3a78bab98d, bc02fc41f5074b338c54e94977f165a9; no voters:
I20260810 21:10:27.957573 14602 raft_consensus.cc:2781] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:10:27.957635 14602 raft_consensus.cc:482] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:10:27.957652 14602 raft_consensus.cc:3037] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:27.958397 14602 raft_consensus.cc:504] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.958559 14602 leader_election.cc:288] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [CANDIDATE]: Term 1 election: Requested vote from peers 063bc4065d254c488c13947879ff12a4 (127.27.117.66:42857), b8d874cb6a6c4dbab8d98c3a78bab98d (127.27.117.65:44839)
I20260810 21:10:27.958730 14020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e74cac343d74cebb15a91b8e9c4689b" candidate_uuid: "bc02fc41f5074b338c54e94977f165a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d"
I20260810 21:10:27.958770 14248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e74cac343d74cebb15a91b8e9c4689b" candidate_uuid: "bc02fc41f5074b338c54e94977f165a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063bc4065d254c488c13947879ff12a4"
I20260810 21:10:27.958830 14020 raft_consensus.cc:3037] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:27.958863 14248 raft_consensus.cc:3037] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:10:27.959550 14020 raft_consensus.cc:2454] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc02fc41f5074b338c54e94977f165a9 in term 1.
I20260810 21:10:27.959607 14248 raft_consensus.cc:2454] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc02fc41f5074b338c54e94977f165a9 in term 1.
I20260810 21:10:27.959769 14368 leader_election.cc:302] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8d874cb6a6c4dbab8d98c3a78bab98d, bc02fc41f5074b338c54e94977f165a9; no voters:
I20260810 21:10:27.959873 14602 raft_consensus.cc:2781] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:10:27.960016 14602 raft_consensus.cc:686] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 1 LEADER]: Becoming Leader. State: Replica: bc02fc41f5074b338c54e94977f165a9, State: Running, Role: LEADER
I20260810 21:10:27.960120 14602 consensus_queue.cc:237] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } }
I20260810 21:10:27.960970 13759 catalog_manager.cc:5495] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc02fc41f5074b338c54e94977f165a9 (127.27.117.67). New cstate: current_term: 1 leader_uuid: "bc02fc41f5074b338c54e94977f165a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc02fc41f5074b338c54e94977f165a9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 41361 } health_report { overall_health: HEALTHY } } }
W20260810 21:10:27.970595 14572 tablet.cc:2369] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:10:27.972601 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.970556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.27.117.67:41361)
0810 21:10:27.970597 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 21:10:27.970599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:10:27.970608 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:10:27.970694 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:10:27.970695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:10:27.970713 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:10:27.970790 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:10:27.970931 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.971338 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.972011 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:10:27.972181 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:10:27.972229 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:10:27.972344 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:10:27.972422 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:10:27.972486 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:10:27.972521 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 21:10:27.972541 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:10:27.974136 14020 raft_consensus.cc:1261] T 0e74cac343d74cebb15a91b8e9c4689b P b8d874cb6a6c4dbab8d98c3a78bab98d [term 1 FOLLOWER]: Refusing update from remote peer bc02fc41f5074b338c54e94977f165a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:27.974177 14248 raft_consensus.cc:1261] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 1 FOLLOWER]: Refusing update from remote peer bc02fc41f5074b338c54e94977f165a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:10:27.974448 14602 consensus_queue.cc:1035] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8d874cb6a6c4dbab8d98c3a78bab98d" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:27.974584 14606 consensus_queue.cc:1035] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063bc4065d254c488c13947879ff12a4" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 42857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:10:27.977186 14614 mvcc.cc:205] Tried to move back new op lower bound from 7317078949779787776 to 7317078949725241344. Current Snapshot: MvccSnapshot[applied={T|T < 7317078949779787776}]
I20260810 21:10:27.979149 13760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:27.979235 13760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:10:27.979959 13760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e84660d16274f27baefccd4df01ea99: Sending DeleteTablet for 3 replicas of tablet 0e74cac343d74cebb15a91b8e9c4689b
I20260810 21:10:27.980278 14219 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e74cac343d74cebb15a91b8e9c4689b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:27 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:44374
I20260810 21:10:27.980500 14628 tablet_replica.cc:330] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: stopping tablet replica
I20260810 21:10:27.980526 14469 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e74cac343d74cebb15a91b8e9c4689b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:27 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60968
I20260810 21:10:27.980651 14628 raft_consensus.cc:2229] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:10:27.980698 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 13872
I20260810 21:10:27.980751 14628 raft_consensus.cc:2258] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:27.980754 13988 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e74cac343d74cebb15a91b8e9c4689b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:10:27 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:45116
I20260810 21:10:27.981515 14628 ts_tablet_manager.cc:1905] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:27.983088 14628 ts_tablet_manager.cc:1918] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:27.983151 14628 log.cc:1196] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/0e74cac343d74cebb15a91b8e9c4689b
I20260810 21:10:27.983501 14628 ts_tablet_manager.cc:1939] T 0e74cac343d74cebb15a91b8e9c4689b P 063bc4065d254c488c13947879ff12a4: Deleting consensus metadata
I20260810 21:10:27.989274 13735 catalog_manager.cc:4841] TS 063bc4065d254c488c13947879ff12a4 (127.27.117.66:42857): tablet 0e74cac343d74cebb15a91b8e9c4689b (table test-table [id=5cf94021c7ac456b8fbd2485b685dd59]) successfully deleted
I20260810 21:10:27.989457 14629 tablet_replica.cc:330] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: stopping tablet replica
I20260810 21:10:27.989563 14629 raft_consensus.cc:2229] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:10:27.989681 14629 raft_consensus.cc:2258] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:10:27.990139 14629 ts_tablet_manager.cc:1905] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:10:27.991698 14629 ts_tablet_manager.cc:1918] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:10:27.991775 14629 log.cc:1196] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/0e74cac343d74cebb15a91b8e9c4689b
I20260810 21:10:27.992074 14629 ts_tablet_manager.cc:1939] T 0e74cac343d74cebb15a91b8e9c4689b P bc02fc41f5074b338c54e94977f165a9: Deleting consensus metadata
I20260810 21:10:27.992458 13734 catalog_manager.cc:4841] TS bc02fc41f5074b338c54e94977f165a9 (127.27.117.67:41361): tablet 0e74cac343d74cebb15a91b8e9c4689b (table test-table [id=5cf94021c7ac456b8fbd2485b685dd59]) successfully deleted
W20260810 21:10:27.994148 13734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:44839 (error 108)
I20260810 21:10:27.994439 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 14113
I20260810 21:10:27.998121 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:10:27.994541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:44839 (local address 127.0.0.1:45120)
0810 21:10:27.998031 (+ 3490us) negotiation.cc:107] Waiting for socket to connect
0810 21:10:27.998081 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:44839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3463}
W20260810 21:10:27.998495 13734 catalog_manager.cc:4584] TS b8d874cb6a6c4dbab8d98c3a78bab98d (127.27.117.65:44839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e74cac343d74cebb15a91b8e9c4689b: Network error: Client connection negotiation failed: client connection to 127.27.117.65:44839: connect: Connection refused (error 111)
I20260810 21:10:28.012339 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 14336
I20260810 21:10:28.018961 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 13707
2026-08-10T21:10:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2328 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:10:28 dist-test-slave-2kcd krb5kdc[14641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:10:28 dist-test-slave-2kcd krb5kdc[14641](info): set up 2 sockets
Aug 10 21:10:28 dist-test-slave-2kcd krb5kdc[14641](info): commencing operation
krb5kdc: starting...
W20260810 21:10:29.290503 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:10:29 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:10:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/rangeradmin_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/rangeradmin_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/rangerlookup_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/rangerlookup_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
I20260810 21:10:29.447377 28117 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksacWUR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/postgres -l logfile start
2026-08-10 21:10:31.792 UTC [14689] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 21:10:31.792 UTC [14689] LOG: listening on IPv4 address "127.27.117.84", port 37729
2026-08-10 21:10:31.795 UTC [14689] LOG: listening on Unix socket "/tmp/.s.PGSQL.37729"
2026-08-10 21:10:31.831 UTC [14694] LOG: database system was shut down at 2026-08-10 21:10:30 UTC
2026-08-10 21:10:31.835 UTC [14689] LOG: database system is ready to accept connections
I20260810 21:10:33.035442 28117 mini_postgres.cc:96] Postgres bound to 37729
2026-08-10 21:10:33.044 UTC [14710] FATAL: database "slave" does not exist
127.27.117.84:37729 - accepting connections
I20260810 21:10:33.044936 28117 mini_ranger.cc:161] Starting Ranger...
I20260810 21:10:33.061853 28117 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 21:10:33.486976 28117 mini_ranger.cc:87] Created ranger Postgres database
I20260810 21:10:33.487074 28117 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-admin
2026-08-10 21:10:33,868 [I] DB FLAVOR :POSTGRES
2026-08-10 21:10:33,869 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 21:10:33,869 [I] Checking connection..
2026-08-10 21:10:33,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 21:10:34,136 [I] Checking connection passed.
2026-08-10 21:10:34,137 [I] --------- Verifying version history table ---------
2026-08-10 21:10:34,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 21:10:34,408 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 21:10:34,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 21:10:34,666 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 21:10:34,666 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 21:10:34,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 21:10:34.903 UTC [14830] WARNING: there is no transaction in progress
2026-08-10 21:10:34,912 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 21:10:34,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 21:10:35,179 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 21:10:35,179 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 21:10:35,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 21:10:35,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 21:10:35,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2kcd.c.gcp-upstream.internal','N') ;"
2026-08-10 21:10:35,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 21:10:36,171 [I] Table x_portal_user does not exist in database ranger
2026-08-10 21:10:36,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 21:10:36,424 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 21:10:36,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 21:10:36,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 21:10:36,940 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 21:10:36,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 21:10:37.853 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.869 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.884 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.898 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.912 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.928 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.981 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:37.988 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.001 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.011 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.020 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.033 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.042 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.048 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.054 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38.060 UTC [15020] WARNING: there is no transaction in progress
2026-08-10 21:10:38,627 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 21:10:38,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 21:10:38,864 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 21:10:38,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 21:10:39,131 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 21:10:39,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 21:10:39,389 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 21:10:39,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 21:10:39,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 21:10:39,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2kcd.c.gcp-upstream.internal';"
2026-08-10 21:10:40,161 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 21:10:40,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 21:10:40,410 [I] DB_PATCHES have already been applied
I20260810 21:10:40.416133 28117 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksacWUR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 21:10:40.416221 28117 mini_ranger.cc:193] Using host: 127.27.117.84
I20260810 21:10:40.419982 28117 mini_ranger.cc:227] Ranger admin URL: http://127.27.117.84:34949
Aug 10, 2026 9:10:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 9:10:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 9:10:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 9:10:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 9:10:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 9:10:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 9:10:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 21:10:41.875690 28117 mini_ranger.cc:160] Time spent starting Ranger: real 8.831s user 0.000s sys 0.011s
Aug 10, 2026 9:11:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 9:11:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 9:11:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 9:11:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 9:11:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 21:11:04.231583 28117 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/rangerkms_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/rangerkms_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.117.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.117.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 21:11:04 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396264, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 21:11:04.316942 28117 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 21:11:04.327834 28117 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 21:11:05.068166 28117 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 21:11:05.068238 28117 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms
I20260810 21:11:05.068250 28117 mini_ranger_kms.cc:227] Using postgres at 127.27.117.84:37729
2026-08-10 21:11:06,067 [I] DB FLAVOR :POSTGRES
2026-08-10 21:11:06,067 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 21:11:06,068 [I] Checking connection
2026-08-10 21:11:06,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 21:11:06,392 [I] connection success
2026-08-10 21:11:06,392 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 21:11:06,392 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 21:11:06,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 21:11:06,664 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 21:11:06,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 21:11:06,665 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 21:11:06,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 21:11:06,916 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 21:11:07.401237 28117 mini_ranger_kms.cc:310] Created kms service
I20260810 21:11:07.588234 28117 mini_ranger_kms.cc:326] Created kudu user
I20260810 21:11:07.634548 28117 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 21:11:07 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396264, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:11:08.046114 28117 mini_ranger_kms.cc:384] Added ranger policy
I20260810 21:11:08.046412 28117 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 21:11:08.046442 28117 mini_ranger_kms.cc:240] Using host: 127.27.117.84
I20260810 21:11:08.051093 28117 mini_ranger_kms.cc:276] Ranger KMS PID: 15334
I20260810 21:11:08.051208 28117 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.117.84:38763
Aug 10, 2026 9:11:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 9:11:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 9:11:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 9:11:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 9:11:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 9:11:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 21:11:10 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786396270, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.117.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:11:10.886634 28117 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.570s user 0.000s sys 0.726s
I20260810 21:11:10.886762 28117 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 21:11:10.886847 28117 mini_ranger_kms.cc:401] 127.27.117.84:38763/kms/v1/keys
Aug 10 21:11:11 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396270, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.117.84@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
Aug 10 21:11:11 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396270, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.117.84@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
Aug 10 21:11:14 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:11:14.071748 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40469
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:40469
--ranger_config_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:38763/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:11:14.178015 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:11:14.178218 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:11:14.178238 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:11:14.179378 15428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:11:14.179418 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:11:14.179431 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:11:14.179443 15428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:11:14.179454 15428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:11:14.181186 15428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.117.84:38763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:40469
--ranger_config_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:11:14.181418 15428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:11:14.181666 15428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:11:14.187441 15435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:14.187523 15434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:14.187569 15437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:11:14.187925 15428 server_base.cc:1034] running on GCE node
I20260810 21:11:14.188201 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:11:14.188526 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:11:14.189733 15428 hybrid_clock.cc:648] HybridClock initialized: now 1786396274189697 us; error 52 us; skew 500 ppm
Aug 10 21:11:14 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396274, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:11:14.192973 15428 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:11:14.193415 15428 webserver.cc:460] Webserver started at http://127.27.117.126:42159/ using document root <none> and password file <none>
I20260810 21:11:14.193651 15428 fs_manager.cc:359] Metadata directory not provided
I20260810 21:11:14.193712 15428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:11:14.193872 15428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:11:14 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396274, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:11:14.241959 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "04b098c46b934ea78a62ebaee5e9ce85"
format_stamp: "Formatted at 2026-08-10 21:11:14 on dist-test-slave-2kcd"
server_key: "549c46d454f830e352bd4c87442efa21"
server_key_iv: "89f83432ce130449ea49fd7ac91a5291"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:14.242413 15428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "04b098c46b934ea78a62ebaee5e9ce85"
format_stamp: "Formatted at 2026-08-10 21:11:14 on dist-test-slave-2kcd"
server_key: "549c46d454f830e352bd4c87442efa21"
server_key_iv: "89f83432ce130449ea49fd7ac91a5291"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:14.244073 15428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:11:14.261263 15458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:14.261673 15428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 21:11:14.261812 15428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "04b098c46b934ea78a62ebaee5e9ce85"
format_stamp: "Formatted at 2026-08-10 21:11:14 on dist-test-slave-2kcd"
server_key: "549c46d454f830e352bd4c87442efa21"
server_key_iv: "89f83432ce130449ea49fd7ac91a5291"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:14.261932 15428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:11:14.276727 15428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:11:14.279011 15428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:11:14.279235 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:11:14.287510 15428 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:40469
I20260810 21:11:14.287621 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:40469 every 8 connection(s)
I20260810 21:11:14.288002 15428 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:11:14.288924 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:11:14.291812 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85: Bootstrap starting.
I20260810 21:11:14.291957 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 15428
I20260810 21:11:14.292080 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:11:14.292279 28117 external_mini_cluster.cc:1428] Setting key 7eb66cfe7ed21ac9789766ad6e04d00b
I20260810 21:11:14.292979 15549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85: Neither blocks nor log segments found. Creating new log.
I20260810 21:11:14.293678 15549 log.cc:824] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85: Log is configured to *not* fsync() on all Append() calls
I20260810 21:11:14.294601 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85: No bootstrap required, opened a new log
I20260810 21:11:14.298295 15549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b098c46b934ea78a62ebaee5e9ce85" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40469 } }
I20260810 21:11:14.298502 15549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:11:14.298545 15549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b098c46b934ea78a62ebaee5e9ce85, State: Initialized, Role: FOLLOWER
I20260810 21:11:14.298698 15549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b098c46b934ea78a62ebaee5e9ce85" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40469 } }
I20260810 21:11:14.298775 15549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:11:14.298799 15549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:11:14.298942 15549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:11:14.299659 15549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b098c46b934ea78a62ebaee5e9ce85" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40469 } }
I20260810 21:11:14.299748 15549 leader_election.cc:302] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04b098c46b934ea78a62ebaee5e9ce85; no voters:
I20260810 21:11:14.299937 15549 leader_election.cc:288] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:11:14.300060 15557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 1 FOLLOWER]: Leader election won for term 1
Aug 10 21:11:14 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:11:14.301054 15549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:11:14.301074 15557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [term 1 LEADER]: Becoming Leader. State: Replica: 04b098c46b934ea78a62ebaee5e9ce85, State: Running, Role: LEADER
I20260810 21:11:14.301172 15557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b098c46b934ea78a62ebaee5e9ce85" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40469 } }
I20260810 21:11:14.301723 15558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04b098c46b934ea78a62ebaee5e9ce85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b098c46b934ea78a62ebaee5e9ce85" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40469 } } }
I20260810 21:11:14.301739 15559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04b098c46b934ea78a62ebaee5e9ce85. Latest consensus state: current_term: 1 leader_uuid: "04b098c46b934ea78a62ebaee5e9ce85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04b098c46b934ea78a62ebaee5e9ce85" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40469 } } }
I20260810 21:11:14.301941 15559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [sys.catalog]: This master's current role is: LEADER
I20260810 21:11:14.301929 15558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85 [sys.catalog]: This master's current role is: LEADER
I20260810 21:11:14.302716 15566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:11:14.303864 15566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:11:14.304611 15549 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 21:11:14.305917 15566 catalog_manager.cc:1347] Generated new cluster ID: 503f85b35bac4747a75a562ce37b9951
I20260810 21:11:14.305969 15566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:11:14.306552 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:14.292878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49072 (local address 127.27.117.126:40469)
0810 21:11:14.293221 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:14.293230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:14.293264 (+ 34us) server_negotiation.cc:407] Connection header received
0810 21:11:14.293912 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:14.293919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:14.293967 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:14.294086 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:11:14.294910 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:14.296747 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:14.297562 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:14.298416 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:14.298548 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:14.301447 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:11:14.301451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:11:14.301452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:11:14.301463 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:11:14.304591 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:14.305068 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:14.305071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:14.305071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:14.305228 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:14.305516 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:14.305517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:14.305517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:14.305860 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:11:14.306125 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:14.306221 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 21:11:14.306328 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":161,"threads_started":1}
I20260810 21:11:14.322044 15566 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:11:14.322579 15566 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:11:14.336328 15566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85: Generated new TSK 0
I20260810 21:11:14.336575 15566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:11:15.187711 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:38763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40469
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:11:15.316074 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:11:15.316303 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:11:15.316324 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:11:15.317528 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:11:15.317569 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:11:15.317592 15638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:11:15.319124 15638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.117.84:38763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:11:15.319341 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:11:15.319581 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:11:15.327154 15645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:15.327184 15644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:15.327551 15648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:11:15.328135 15638 server_base.cc:1034] running on GCE node
I20260810 21:11:15.328374 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:11:15.328706 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:11:15.329891 15638 hybrid_clock.cc:648] HybridClock initialized: now 1786396275329854 us; error 51 us; skew 500 ppm
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:11:15.333739 15638 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:11:15.334234 15638 webserver.cc:460] Webserver started at http://127.27.117.65:46343/ using document root <none> and password file <none>
I20260810 21:11:15.334477 15638 fs_manager.cc:359] Metadata directory not provided
I20260810 21:11:15.334547 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:11:15.334688 15638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:11:15.351110 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "424a8f07fcf641fd9be941404c9d2c79"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "b05f5c47696b508678c952b829d2c00e"
server_key_iv: "d749b16a9e4a900f9033a18b7a69396c"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.351750 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "424a8f07fcf641fd9be941404c9d2c79"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "b05f5c47696b508678c952b829d2c00e"
server_key_iv: "d749b16a9e4a900f9033a18b7a69396c"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.355718 15638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 21:11:15.370226 15657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.370398 15638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:11:15.370497 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "424a8f07fcf641fd9be941404c9d2c79"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "b05f5c47696b508678c952b829d2c00e"
server_key_iv: "d749b16a9e4a900f9033a18b7a69396c"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.370584 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:11:15.410542 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:11:15.412750 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:11:15.412930 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:11:15.413291 15638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:11:15.413755 15638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:11:15.413812 15638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.413853 15638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:11:15.413880 15638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.430126 15638 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37083
I20260810 21:11:15.430212 15883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37083 every 8 connection(s)
I20260810 21:11:15.430697 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:11:15.439016 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 15638
I20260810 21:11:15.439141 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:11:15.439339 28117 external_mini_cluster.cc:1428] Setting key 9a75766d43417aac52e3789203f8ea24
I20260810 21:11:15.442781 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:15.432256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:41031 (local address 127.27.117.126:40469)
0810 21:11:15.432423 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:15.432425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:15.432451 (+ 26us) server_negotiation.cc:407] Connection header received
0810 21:11:15.433200 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:15.433202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:15.433217 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:15.433326 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:11:15.434112 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.435069 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.436121 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.436472 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.436564 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:15.439110 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:11:15.439112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:11:15.439112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:11:15.439119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:11:15.441029 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:15.441488 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:15.441488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:15.441489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:15.441556 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:15.441812 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:15.441813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:15.441813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:15.441937 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:11:15.442008 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:15.442584 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 21:11:15.442667 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260810 21:11:15.442960 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:15.431884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40469 (local address 127.27.117.65:41031)
0810 21:11:15.432187 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:15.432233 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:15.433057 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:15.433400 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:15.433404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:15.433743 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:11:15.433999 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:15.434004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.435166 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.435166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:15.436030 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:15.436032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.436569 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.436570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:15.436761 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:15.437441 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:11:15.437448 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:11:15.439012 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:11:15.441122 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:11:15.441128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:11:15.441130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:11:15.441402 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:11:15.441624 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:11:15.441625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:11:15.441626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:11:15.441745 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:11:15.442036 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:11:15.442042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:11:15.442423 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 21:11:15.442547 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 21:11:15.442695 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":186,"threads_started":1}
I20260810 21:11:15.443624 15884 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40469
I20260810 21:11:15.443706 15884 heartbeater.cc:464] Registering TS with master...
I20260810 21:11:15.443879 15884 heartbeater.cc:511] Master 127.27.117.126:40469 requested a full tablet report, sending...
I20260810 21:11:15.444471 15482 ts_manager.cc:194] Registered new tserver with Master: 424a8f07fcf641fd9be941404c9d2c79 (127.27.117.65:37083)
I20260810 21:11:15.445573 15482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:41031
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:11:15.469472 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:38763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40469
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:11:15.607466 15896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:11:15.607712 15896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:11:15.607752 15896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:11:15.609002 15896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:11:15.609064 15896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:11:15.609108 15896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:11:15.610864 15896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.117.84:38763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:11:15.611128 15896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:11:15.611397 15896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:11:15.618275 15921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:15.618451 15910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:15.618319 15912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:11:15.619565 15896 server_base.cc:1034] running on GCE node
I20260810 21:11:15.619807 15896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:11:15.620098 15896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:11:15.621253 15896 hybrid_clock.cc:648] HybridClock initialized: now 1786396275621245 us; error 32 us; skew 500 ppm
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:11:15.624326 15896 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:11:15.624893 15896 webserver.cc:460] Webserver started at http://127.27.117.66:45151/ using document root <none> and password file <none>
I20260810 21:11:15.625178 15896 fs_manager.cc:359] Metadata directory not provided
I20260810 21:11:15.625250 15896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:11:15.625394 15896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:11:15.644767 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "86d7e2ad1b524a48846771213c79622c"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "836aa74b6b9629d521f2e2f671737b82"
server_key_iv: "d717c5219e1a7e9f4dba919ddb1dce81"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.645380 15896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "86d7e2ad1b524a48846771213c79622c"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "836aa74b6b9629d521f2e2f671737b82"
server_key_iv: "d717c5219e1a7e9f4dba919ddb1dce81"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.647559 15896 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 21:11:15.666886 15936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.667054 15896 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 21:11:15.667140 15896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "86d7e2ad1b524a48846771213c79622c"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "836aa74b6b9629d521f2e2f671737b82"
server_key_iv: "d717c5219e1a7e9f4dba919ddb1dce81"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.667224 15896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:11:15.689086 15896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:11:15.691479 15896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:11:15.691627 15896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:11:15.691886 15896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:11:15.692225 15896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:11:15.692262 15896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.692291 15896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:11:15.692317 15896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.705585 15896 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:38591
I20260810 21:11:15.706135 15896 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:11:15.707526 16143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:38591 every 8 connection(s)
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:11:15.717025 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 15896
I20260810 21:11:15.717144 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:11:15.717360 28117 external_mini_cluster.cc:1428] Setting key a9408d6141bc03ff0bd8c8dc5b5951a8
I20260810 21:11:15.722083 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:15.706926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:38845 (local address 127.27.117.126:40469)
0810 21:11:15.706976 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:15.706979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:15.707292 (+ 313us) server_negotiation.cc:407] Connection header received
0810 21:11:15.708053 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:15.708054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:15.708072 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:15.708137 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:11:15.709052 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.709834 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.711469 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.711864 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.711933 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:15.714797 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:11:15.714798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:11:15.714799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:11:15.714807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:11:15.716743 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:15.720601 (+ 3858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:15.720611 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:15.720612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:15.720676 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:15.721157 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:15.721158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:15.721159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:15.721295 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:11:15.721339 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:15.721882 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 21:11:15.721987 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:11:15.722884 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:15.706919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40469 (local address 127.27.117.66:38845)
0810 21:11:15.707160 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:15.707205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:15.707931 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:15.708209 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:15.708213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:15.708591 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:11:15.708947 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:15.708953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.709953 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.709955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:15.711361 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:15.711364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:15.711975 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:15.711976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:15.712163 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:15.712817 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:11:15.712835 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:11:15.714692 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:11:15.717686 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:11:15.717692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:11:15.717694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:11:15.718001 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:11:15.720761 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:11:15.720763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:11:15.720763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:11:15.721052 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:11:15.721389 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:11:15.721395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:11:15.721697 (+ 302us) client_negotiation.cc:769] Sending connection context
0810 21:11:15.722460 (+ 763us) client_negotiation.cc:240] Negotiation successful
0810 21:11:15.722613 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":157,"threads_started":1}
I20260810 21:11:15.723596 16146 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40469
I20260810 21:11:15.723739 16146 heartbeater.cc:464] Registering TS with master...
I20260810 21:11:15.723974 16146 heartbeater.cc:511] Master 127.27.117.126:40469 requested a full tablet report, sending...
I20260810 21:11:15.724339 15482 ts_manager.cc:194] Registered new tserver with Master: 86d7e2ad1b524a48846771213c79622c (127.27.117.66:38591)
I20260810 21:11:15.724856 15482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:38845
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:11:15.758751 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:38763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40469
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:11:15.912957 16154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:11:15.913329 16154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:11:15.913419 16154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:11:15.915340 16154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:11:15.915455 16154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:11:15.915539 16154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:11:15.917938 16154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.117.84:38763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:11:15.918417 16154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:11:15.918748 16154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:11:15.927338 16171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:15.927711 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:11:15.927959 16173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:11:15.929234 16154 server_base.cc:1034] running on GCE node
I20260810 21:11:15.929572 16154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:11:15.929854 16154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:11:15.931010 16154 hybrid_clock.cc:648] HybridClock initialized: now 1786396275930989 us; error 33 us; skew 500 ppm
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:11:15.934798 16154 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:11:15.935343 16154 webserver.cc:460] Webserver started at http://127.27.117.67:43703/ using document root <none> and password file <none>
I20260810 21:11:15.935634 16154 fs_manager.cc:359] Metadata directory not provided
I20260810 21:11:15.935685 16154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:11:15.935808 16154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:11:15 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:11:15.960016 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "e930d95a55b44aaeb2e0989655d27482"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "8c3dac3b6d5652f1f242c982277b40f4"
server_key_iv: "6a398be5787643208595f6aab3e3ddd2"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.960690 16154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "e930d95a55b44aaeb2e0989655d27482"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "8c3dac3b6d5652f1f242c982277b40f4"
server_key_iv: "6a398be5787643208595f6aab3e3ddd2"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.964772 16154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 21:11:15.991364 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:15.991647 16154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:11:15.991794 16154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "e930d95a55b44aaeb2e0989655d27482"
format_stamp: "Formatted at 2026-08-10 21:11:15 on dist-test-slave-2kcd"
server_key: "8c3dac3b6d5652f1f242c982277b40f4"
server_key_iv: "6a398be5787643208595f6aab3e3ddd2"
server_key_version: "kuduclusterkey@0"
I20260810 21:11:15.991977 16154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:11:16.006819 16154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:11:16.009372 16154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:11:16.009539 16154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:11:16.009879 16154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:11:16.010326 16154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:11:16.010385 16154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:16.010425 16154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:11:16.010449 16154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:11:16.025882 16154 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:42481
I20260810 21:11:16.026439 16154 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:11:16.033036 16379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:42481 every 8 connection(s)
I20260810 21:11:16.033277 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 16154
I20260810 21:11:16.033365 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:11:16.033569 28117 external_mini_cluster.cc:1428] Setting key a6178611477c78dbd868e3a80d516ade
Aug 10 21:11:16 dist-test-slave-2kcd krb5kdc[14641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:11:16.043159 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.034070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53269 (local address 127.27.117.126:40469)
0810 21:11:16.034141 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.034144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.034314 (+ 170us) server_negotiation.cc:407] Connection header received
0810 21:11:16.035106 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.035107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.035119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.035175 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:11:16.035933 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.036431 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.037502 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.037694 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.037771 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.040135 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:11:16.040137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:11:16.040138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:11:16.040184 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:11:16.041694 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:16.042139 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:16.042140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:16.042140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:16.042199 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:16.042449 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:16.042450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:16.042450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:16.042583 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:11:16.042656 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.042972 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.043070 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:11:16.043366 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.033779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40469 (local address 127.27.117.67:53269)
0810 21:11:16.034160 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.034210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.034990 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.035251 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.035254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.035560 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:11:16.035830 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.035835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.036526 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.036527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.037413 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.037416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.037776 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.037778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.037894 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.038545 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:11:16.038554 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:11:16.040011 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:11:16.041799 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:11:16.041806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:11:16.041808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:11:16.042036 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:11:16.042266 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:11:16.042266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:11:16.042267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:11:16.042381 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:11:16.042658 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:11:16.042684 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:11:16.042824 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.042935 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.043083 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":82,"threads_started":1}
I20260810 21:11:16.043984 16380 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40469
I20260810 21:11:16.044109 16380 heartbeater.cc:464] Registering TS with master...
I20260810 21:11:16.044292 16380 heartbeater.cc:511] Master 127.27.117.126:40469 requested a full tablet report, sending...
I20260810 21:11:16.044698 15482 ts_manager.cc:194] Registered new tserver with Master: e930d95a55b44aaeb2e0989655d27482 (127.27.117.67:42481)
I20260810 21:11:16.045133 15482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:53269
I20260810 21:11:16.045708 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:11:16.053661 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.047209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.27.117.126:40469)
0810 21:11:16.047248 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.047250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.047422 (+ 172us) server_negotiation.cc:407] Connection header received
0810 21:11:16.047493 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.047494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.047504 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.047559 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:11:16.048162 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.048641 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.049483 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.049680 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.049755 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.051032 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:11:16.051033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:11:16.051034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:11:16.051040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:11:16.052480 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:16.052879 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:16.052880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:16.052881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:16.052935 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:11:16.053142 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:11:16.053142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:11:16.053143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:11:16.053251 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:11:16.053353 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.053481 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.053556 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:11:16.055508 15481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:11:16.057370 15481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:11:16.066299 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.060083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37083 (local address 127.0.0.1:36964)
0810 21:11:16.061167 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.061177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.061238 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.061617 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.061620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.061622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.061686 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.061688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.062632 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.062633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.064476 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.064478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.065614 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.065615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.065716 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.065728 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.066141 (+ 413us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.066219 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":184,"threads_started":1}
I20260810 21:11:16.066673 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.060445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.27.117.65:37083)
0810 21:11:16.061073 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.061079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.061319 (+ 240us) server_negotiation.cc:407] Connection header received
0810 21:11:16.061360 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.061363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.061380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.061483 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.061775 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.062539 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.064572 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.065451 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.065501 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.065558 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.066515 (+ 957us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.066564 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":88,"threads_started":1}
I20260810 21:11:16.067490 15763 tablet_service.cc:1467] Processing CreateTablet for tablet 4be5be699fb3415c859c8018ca3ca165 (DEFAULT_TABLE table=test-table [id=1f01afa97a0846a28ac9d64dec6eae3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:11:16.067682 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.060080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:38591 (local address 127.0.0.1:59010)
0810 21:11:16.060723 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.060765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.060851 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.063133 (+ 2282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.063134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.063136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.063209 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.063211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.064758 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.064759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.065885 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.065887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.067422 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.067423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.067496 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.067501 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.067574 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.067613 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":154,"threads_started":1}
I20260810 21:11:16.067889 15763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be5be699fb3415c859c8018ca3ca165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:11:16.068400 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.060903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.27.117.66:38591)
0810 21:11:16.062960 (+ 2057us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.062965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.062975 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:11:16.063015 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.063017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.063032 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.063289 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.063396 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.064667 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.066084 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.067321 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.067418 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.067554 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.067585 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.067620 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2019,"thread_start_us":159,"threads_started":1}
I20260810 21:11:16.069173 16016 tablet_service.cc:1467] Processing CreateTablet for tablet 4be5be699fb3415c859c8018ca3ca165 (DEFAULT_TABLE table=test-table [id=1f01afa97a0846a28ac9d64dec6eae3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:11:16.069588 16016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be5be699fb3415c859c8018ca3ca165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:11:16.073853 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.060438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42481 (local address 127.0.0.1:47968)
0810 21:11:16.062728 (+ 2290us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.062739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.062785 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.068201 (+ 5416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.068201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.068203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.068257 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.068258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.070337 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.070338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.071820 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.071824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.073550 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.073552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.073647 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.073651 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.073696 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.073742 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2258,"thread_start_us":121,"threads_started":1}
I20260810 21:11:16.074595 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.061353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.27.117.67:42481)
0810 21:11:16.068014 (+ 6661us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.068019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.068030 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:11:16.068080 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.068082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.068099 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.068190 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.068332 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.070214 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.072502 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.073432 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.074206 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.074368 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.074396 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.074442 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6593,"thread_start_us":127,"threads_started":1}
I20260810 21:11:16.074981 16418 tablet_bootstrap.cc:492] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Bootstrap starting.
I20260810 21:11:16.075825 16253 tablet_service.cc:1467] Processing CreateTablet for tablet 4be5be699fb3415c859c8018ca3ca165 (DEFAULT_TABLE table=test-table [id=1f01afa97a0846a28ac9d64dec6eae3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:11:16.076324 16418 tablet_bootstrap.cc:654] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Neither blocks nor log segments found. Creating new log.
I20260810 21:11:16.076458 16253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be5be699fb3415c859c8018ca3ca165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:11:16.077152 16418 log.cc:824] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Log is configured to *not* fsync() on all Append() calls
I20260810 21:11:16.078023 16418 tablet_bootstrap.cc:492] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: No bootstrap required, opened a new log
I20260810 21:11:16.078106 16418 ts_tablet_manager.cc:1397] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:11:16.080421 16418 raft_consensus.cc:348] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.080579 16418 raft_consensus.cc:374] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:11:16.080623 16418 raft_consensus.cc:729] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86d7e2ad1b524a48846771213c79622c, State: Initialized, Role: FOLLOWER
I20260810 21:11:16.080758 16418 consensus_queue.cc:260] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.081238 16146 heartbeater.cc:503] Master 127.27.117.126:40469 was elected leader, sending a full tablet report...
I20260810 21:11:16.081239 16418 ts_tablet_manager.cc:1428] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:11:16.082777 16414 tablet_bootstrap.cc:492] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: Bootstrap starting.
I20260810 21:11:16.083648 16422 tablet_bootstrap.cc:492] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Bootstrap starting.
I20260810 21:11:16.084229 16414 tablet_bootstrap.cc:654] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: Neither blocks nor log segments found. Creating new log.
I20260810 21:11:16.084973 16414 log.cc:824] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: Log is configured to *not* fsync() on all Append() calls
I20260810 21:11:16.085340 16422 tablet_bootstrap.cc:654] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Neither blocks nor log segments found. Creating new log.
I20260810 21:11:16.086318 16422 log.cc:824] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Log is configured to *not* fsync() on all Append() calls
I20260810 21:11:16.087211 16422 tablet_bootstrap.cc:492] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: No bootstrap required, opened a new log
I20260810 21:11:16.087299 16422 ts_tablet_manager.cc:1397] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 21:11:16.089569 16414 tablet_bootstrap.cc:492] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: No bootstrap required, opened a new log
I20260810 21:11:16.089650 16414 ts_tablet_manager.cc:1397] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260810 21:11:16.090211 16422 raft_consensus.cc:348] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.090368 16422 raft_consensus.cc:374] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:11:16.090416 16422 raft_consensus.cc:729] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e930d95a55b44aaeb2e0989655d27482, State: Initialized, Role: FOLLOWER
I20260810 21:11:16.090559 16422 consensus_queue.cc:260] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.090942 16380 heartbeater.cc:503] Master 127.27.117.126:40469 was elected leader, sending a full tablet report...
I20260810 21:11:16.091199 16422 ts_tablet_manager.cc:1428] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 21:11:16.093766 16414 raft_consensus.cc:348] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.093947 16414 raft_consensus.cc:374] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:11:16.093982 16414 raft_consensus.cc:729] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424a8f07fcf641fd9be941404c9d2c79, State: Initialized, Role: FOLLOWER
I20260810 21:11:16.094113 16414 consensus_queue.cc:260] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.094435 16414 ts_tablet_manager.cc:1428] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260810 21:11:16.094559 15884 heartbeater.cc:503] Master 127.27.117.126:40469 was elected leader, sending a full tablet report...
I20260810 21:11:16.144533 16428 raft_consensus.cc:482] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:11:16.144768 16428 raft_consensus.cc:504] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.145365 16428 leader_election.cc:288] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86d7e2ad1b524a48846771213c79622c (127.27.117.66:38591), e930d95a55b44aaeb2e0989655d27482 (127.27.117.67:42481)
I20260810 21:11:16.149037 16420 raft_consensus.cc:482] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:11:16.149143 16420 raft_consensus.cc:504] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.149624 16420 leader_election.cc:288] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 424a8f07fcf641fd9be941404c9d2c79 (127.27.117.65:37083), e930d95a55b44aaeb2e0989655d27482 (127.27.117.67:42481)
I20260810 21:11:16.151584 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.146336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:58757 (local address 127.27.117.66:38591)
0810 21:11:16.146386 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.146388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.146795 (+ 407us) server_negotiation.cc:407] Connection header received
0810 21:11:16.146937 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.146939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.146953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.146988 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.147847 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.148668 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.150410 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.150975 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.151034 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.151044 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.151468 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.151515 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:11:16.151904 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.145678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:38591 (local address 127.27.117.65:58757)
0810 21:11:16.146710 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.146724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.146842 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.147084 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.147085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.147142 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.147424 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.147428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.148913 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.148914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.150290 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.150293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.151228 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.151229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.151350 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.151356 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.151710 (+ 354us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.151757 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":200,"threads_started":1}
I20260810 21:11:16.152333 16064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be5be699fb3415c859c8018ca3ca165" candidate_uuid: "424a8f07fcf641fd9be941404c9d2c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d7e2ad1b524a48846771213c79622c" is_pre_election: true
I20260810 21:11:16.152508 16064 raft_consensus.cc:2454] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424a8f07fcf641fd9be941404c9d2c79 in term 0.
I20260810 21:11:16.152832 15671 leader_election.cc:302] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424a8f07fcf641fd9be941404c9d2c79, 86d7e2ad1b524a48846771213c79622c; no voters:
I20260810 21:11:16.153035 16428 raft_consensus.cc:2781] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:11:16.153107 16428 raft_consensus.cc:482] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:11:16.153143 16428 raft_consensus.cc:3037] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:11:16.154441 16428 raft_consensus.cc:504] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.154592 16428 leader_election.cc:288] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [CANDIDATE]: Term 1 election: Requested vote from peers 86d7e2ad1b524a48846771213c79622c (127.27.117.66:38591), e930d95a55b44aaeb2e0989655d27482 (127.27.117.67:42481)
I20260810 21:11:16.154881 16064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be5be699fb3415c859c8018ca3ca165" candidate_uuid: "424a8f07fcf641fd9be941404c9d2c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86d7e2ad1b524a48846771213c79622c"
I20260810 21:11:16.154981 16064 raft_consensus.cc:3037] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:11:16.155318 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.146137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:34611 (local address 127.27.117.67:42481)
0810 21:11:16.146186 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.146189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.147662 (+ 1473us) server_negotiation.cc:407] Connection header received
0810 21:11:16.147786 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.147787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.147799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.147873 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.150965 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.151636 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.154201 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.154714 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.154758 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.154766 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.155212 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.155254 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:11:16.155773 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.146022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42481 (local address 127.27.117.65:34611)
0810 21:11:16.147592 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.147603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.147710 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.150638 (+ 2928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.150640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.150642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.150870 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.150872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.152216 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.152217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.154098 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.154101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.154891 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.154892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.155121 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.155126 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.155542 (+ 416us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.155581 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"thread_start_us":126,"threads_started":1}
I20260810 21:11:16.156349 16064 raft_consensus.cc:2454] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424a8f07fcf641fd9be941404c9d2c79 in term 1.
I20260810 21:11:16.156659 15671 leader_election.cc:302] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424a8f07fcf641fd9be941404c9d2c79, 86d7e2ad1b524a48846771213c79622c; no voters:
I20260810 21:11:16.157059 16264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be5be699fb3415c859c8018ca3ca165" candidate_uuid: "424a8f07fcf641fd9be941404c9d2c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930d95a55b44aaeb2e0989655d27482" is_pre_election: true
I20260810 21:11:16.157205 16264 raft_consensus.cc:2454] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424a8f07fcf641fd9be941404c9d2c79 in term 0.
I20260810 21:11:16.157251 16428 raft_consensus.cc:2781] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:11:16.157346 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be5be699fb3415c859c8018ca3ca165" candidate_uuid: "424a8f07fcf641fd9be941404c9d2c79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930d95a55b44aaeb2e0989655d27482"
I20260810 21:11:16.157408 16265 raft_consensus.cc:3037] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:11:16.157473 16428 raft_consensus.cc:686] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [term 1 LEADER]: Becoming Leader. State: Replica: 424a8f07fcf641fd9be941404c9d2c79, State: Running, Role: LEADER
I20260810 21:11:16.157574 16428 consensus_queue.cc:237] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } }
I20260810 21:11:16.158540 16265 raft_consensus.cc:2454] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424a8f07fcf641fd9be941404c9d2c79 in term 1.
I20260810 21:11:16.158510 15481 catalog_manager.cc:5495] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 424a8f07fcf641fd9be941404c9d2c79 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "424a8f07fcf641fd9be941404c9d2c79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "424a8f07fcf641fd9be941404c9d2c79" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 } health_report { overall_health: UNKNOWN } } }
I20260810 21:11:16.162639 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.149837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42481 (local address 127.27.117.66:46619)
0810 21:11:16.149886 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.149901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.149955 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.152743 (+ 2788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.152744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.152746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.152955 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.152957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.160483 (+ 7526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.160484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.161617 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.161620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.162389 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.162390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.162499 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.162504 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.162536 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.162574 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 21:11:16.163125 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.150115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:46619 (local address 127.27.117.67:42481)
0810 21:11:16.152298 (+ 2183us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.152300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.152311 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:11:16.152345 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.152346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.152357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.159502 (+ 7145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.159539 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.160380 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.161719 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.162291 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.162935 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.162944 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.162963 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.162998 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":128,"threads_started":1}
I20260810 21:11:16.163379 16265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be5be699fb3415c859c8018ca3ca165" candidate_uuid: "86d7e2ad1b524a48846771213c79622c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e930d95a55b44aaeb2e0989655d27482" is_pre_election: true
I20260810 21:11:16.163460 16265 raft_consensus.cc:2379] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86d7e2ad1b524a48846771213c79622c in current term 1: Already voted for candidate 424a8f07fcf641fd9be941404c9d2c79 in this term.
I20260810 21:11:16.164079 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.151096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:49699 (local address 127.27.117.65:37083)
0810 21:11:16.151144 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.151146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.152617 (+ 1471us) server_negotiation.cc:407] Connection header received
0810 21:11:16.159315 (+ 6698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.159316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.159329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.159439 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:11:16.160811 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.161523 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.163096 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.163635 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.163683 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.163690 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.163974 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.164022 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:11:16.164179 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.150448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37083 (local address 127.27.117.66:49699)
0810 21:11:16.152549 (+ 2101us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.152561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:11:16.152649 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:11:16.160518 (+ 7869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:11:16.160519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:11:16.160523 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:11:16.160699 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.160701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.161627 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.161628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.162988 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:11:16.162991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.163803 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.163804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:11:16.163892 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.163896 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:11:16.163951 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 21:11:16.163989 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"thread_start_us":108,"threads_started":1}
I20260810 21:11:16.164587 15801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be5be699fb3415c859c8018ca3ca165" candidate_uuid: "86d7e2ad1b524a48846771213c79622c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "424a8f07fcf641fd9be941404c9d2c79" is_pre_election: true
I20260810 21:11:16.164928 15947 leader_election.cc:302] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86d7e2ad1b524a48846771213c79622c; no voters: 424a8f07fcf641fd9be941404c9d2c79, e930d95a55b44aaeb2e0989655d27482
I20260810 21:11:16.165115 16420 raft_consensus.cc:2726] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 21:11:16.186031 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.182310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.27.117.65:37083)
0810 21:11:16.182346 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:11:16.182348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:11:16.182355 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:11:16.182395 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:11:16.182396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:11:16.182415 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:11:16.182475 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:11:16.182643 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.183426 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.184755 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:11:16.185098 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:11:16.185164 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:11:16.185380 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:11:16.185512 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:11:16.185583 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:11:16.185810 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 21:11:16.185829 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 21:11:16.186268 15888 tablet.cc:2369] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:11:16.188843 16265 raft_consensus.cc:1261] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 1 FOLLOWER]: Refusing update from remote peer 424a8f07fcf641fd9be941404c9d2c79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:11:16.189291 16437 consensus_queue.cc:1035] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e930d95a55b44aaeb2e0989655d27482" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:11:16.192094 16064 raft_consensus.cc:1261] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 1 FOLLOWER]: Refusing update from remote peer 424a8f07fcf641fd9be941404c9d2c79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:11:16.192400 16437 consensus_queue.cc:1035] T 4be5be699fb3415c859c8018ca3ca165 P 424a8f07fcf641fd9be941404c9d2c79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86d7e2ad1b524a48846771213c79622c" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 38591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:11:16.196676 16447 mvcc.cc:205] Tried to move back new op lower bound from 7317079147266248704 to 7317079147141967872. Current Snapshot: MvccSnapshot[applied={T|T < 7317079147266248704}]
I20260810 21:11:16.197041 16451 mvcc.cc:205] Tried to move back new op lower bound from 7317079147266248704 to 7317079147141967872. Current Snapshot: MvccSnapshot[applied={T|T < 7317079147266248704}]
I20260810 21:11:16.202343 15481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:11:16.202473 15481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:11:16.203470 15481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04b098c46b934ea78a62ebaee5e9ce85: Sending DeleteTablet for 3 replicas of tablet 4be5be699fb3415c859c8018ca3ca165
I20260810 21:11:16.204126 16016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be5be699fb3415c859c8018ca3ca165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:11:16 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:59010
I20260810 21:11:16.204499 16253 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be5be699fb3415c859c8018ca3ca165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:11:16 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:47968
I20260810 21:11:16.205106 15763 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be5be699fb3415c859c8018ca3ca165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:11:16 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:36964
I20260810 21:11:16.205708 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 15638
I20260810 21:11:16.223523 16449 mvcc.cc:205] Tried to move back new op lower bound from 7317079147266248704 to 7317079147141967872. Current Snapshot: MvccSnapshot[applied={T|T < 7317079147266248704}]
I20260810 21:11:16.225010 16463 tablet_replica.cc:330] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: stopping tablet replica
I20260810 21:11:16.225253 16463 raft_consensus.cc:2229] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:11:16.225479 16463 raft_consensus.cc:2258] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:11:16.225811 16464 tablet_replica.cc:330] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: stopping tablet replica
I20260810 21:11:16.225921 16464 raft_consensus.cc:2229] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:11:16.226032 16464 raft_consensus.cc:2258] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:11:16.231534 16464 ts_tablet_manager.cc:1905] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:11:16.232317 16463 ts_tablet_manager.cc:1905] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:11:16.234498 16463 ts_tablet_manager.cc:1918] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:11:16.234577 16463 log.cc:1196] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/4be5be699fb3415c859c8018ca3ca165
I20260810 21:11:16.234879 16463 ts_tablet_manager.cc:1939] T 4be5be699fb3415c859c8018ca3ca165 P 86d7e2ad1b524a48846771213c79622c: Deleting consensus metadata
I20260810 21:11:16.235335 15462 catalog_manager.cc:4841] TS 86d7e2ad1b524a48846771213c79622c (127.27.117.66:38591): tablet 4be5be699fb3415c859c8018ca3ca165 (table test-table [id=1f01afa97a0846a28ac9d64dec6eae3c]) successfully deleted
W20260810 21:11:16.235896 15464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:37083 (error 108)
I20260810 21:11:16.236809 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:11:16.236674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37083 (local address 127.0.0.1:36980)
0810 21:11:16.236723 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 21:11:16.236756 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 21:11:16.236924 15464 catalog_manager.cc:4584] TS 424a8f07fcf641fd9be941404c9d2c79 (127.27.117.65:37083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4be5be699fb3415c859c8018ca3ca165: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37083: connect: Connection refused (error 111)
I20260810 21:11:16.236960 16464 ts_tablet_manager.cc:1918] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:11:16.237035 16464 log.cc:1196] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/4be5be699fb3415c859c8018ca3ca165
I20260810 21:11:16.237123 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 15896
I20260810 21:11:16.243706 16464 ts_tablet_manager.cc:1939] T 4be5be699fb3415c859c8018ca3ca165 P e930d95a55b44aaeb2e0989655d27482: Deleting consensus metadata
I20260810 21:11:16.244114 15462 catalog_manager.cc:4841] TS e930d95a55b44aaeb2e0989655d27482 (127.27.117.67:42481): tablet 4be5be699fb3415c859c8018ca3ca165 (table test-table [id=1f01afa97a0846a28ac9d64dec6eae3c]) successfully deleted
I20260810 21:11:16.258054 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 16154
I20260810 21:11:16.287258 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 15428
I20260810 21:11:16.300721 28117 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 21:11:16.649082 28117 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 21:11:16.649187 28117 mini_ranger.cc:66] Stopping Ranger...
I20260810 21:11:16.994123 28117 mini_ranger.cc:68] Stopped Ranger
2026-08-10 21:11:16.994 UTC [14689] LOG: received smart shutdown request
2026-08-10 21:11:16.998 UTC [14689] LOG: background worker "logical replication launcher" (PID 14708) exited with exit code 1
2026-08-10 21:11:16.998 UTC [14695] LOG: shutting down
2026-08-10 21:11:17.139 UTC [14689] LOG: database system is shut down
2026-08-10T21:11:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49229 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:11:17 dist-test-slave-2kcd krb5kdc[16485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:11:17 dist-test-slave-2kcd krb5kdc[16485](info): set up 2 sockets
Aug 10 21:11:17 dist-test-slave-2kcd krb5kdc[16485](info): commencing operation
krb5kdc: starting...
W20260810 21:11:18.536942 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:11:18 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:11:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/rangeradmin_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/rangeradmin_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/rangerlookup_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/rangerlookup_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
I20260810 21:11:18.697371 28117 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksacWUR/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/postgres -l logfile start
2026-08-10 21:11:20.936 UTC [16538] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 21:11:20.936 UTC [16538] LOG: listening on IPv4 address "127.27.117.84", port 37603
2026-08-10 21:11:20.940 UTC [16538] LOG: listening on Unix socket "/tmp/.s.PGSQL.37603"
2026-08-10 21:11:20.960 UTC [16544] LOG: database system was shut down at 2026-08-10 21:11:19 UTC
2026-08-10 21:11:20.964 UTC [16538] LOG: database system is ready to accept connections
I20260810 21:11:22.157821 28117 mini_postgres.cc:96] Postgres bound to 37603
2026-08-10 21:11:22.165 UTC [16560] FATAL: database "slave" does not exist
127.27.117.84:37603 - accepting connections
I20260810 21:11:22.165568 28117 mini_ranger.cc:161] Starting Ranger...
I20260810 21:11:22.180886 28117 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 21:11:22.571362 28117 mini_ranger.cc:87] Created ranger Postgres database
I20260810 21:11:22.571434 28117 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-admin
2026-08-10 21:11:22,847 [I] DB FLAVOR :POSTGRES
2026-08-10 21:11:22,848 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 21:11:22,848 [I] Checking connection..
2026-08-10 21:11:22,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 21:11:23,116 [I] Checking connection passed.
2026-08-10 21:11:23,117 [I] --------- Verifying version history table ---------
2026-08-10 21:11:23,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 21:11:23,384 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 21:11:23,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 21:11:23,657 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 21:11:23,658 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 21:11:23,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 21:11:23.907 UTC [16676] WARNING: there is no transaction in progress
2026-08-10 21:11:23,917 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 21:11:23,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 21:11:24,177 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 21:11:24,178 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 21:11:24,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 21:11:24,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 21:11:24,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2kcd.c.gcp-upstream.internal','N') ;"
2026-08-10 21:11:24,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 21:11:25,163 [I] Table x_portal_user does not exist in database ranger
2026-08-10 21:11:25,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 21:11:25,425 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 21:11:25,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 21:11:25,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 21:11:25,916 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 21:11:25,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 21:11:26.829 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.846 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.860 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.874 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.892 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.911 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.964 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.971 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.984 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:26.993 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27.003 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27.015 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27.024 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27.030 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27.036 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27.042 UTC [16870] WARNING: there is no transaction in progress
2026-08-10 21:11:27,626 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 21:11:27,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 21:11:27,876 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 21:11:27,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 21:11:28,121 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 21:11:28,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 21:11:28,383 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 21:11:28,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 21:11:28,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 21:11:28,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2kcd.c.gcp-upstream.internal';"
2026-08-10 21:11:29,145 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 21:11:29,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 21:11:29,401 [I] DB_PATCHES have already been applied
I20260810 21:11:29.406937 28117 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksacWUR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 21:11:29.407001 28117 mini_ranger.cc:193] Using host: 127.27.117.84
I20260810 21:11:29.410765 28117 mini_ranger.cc:227] Ranger admin URL: http://127.27.117.84:43795
Aug 10, 2026 9:11:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 9:11:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 9:11:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 9:11:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 9:11:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 9:11:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 9:11:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 21:11:30.819531 28117 mini_ranger.cc:160] Time spent starting Ranger: real 8.654s user 0.000s sys 0.010s
Aug 10, 2026 9:11:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 9:11:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 9:11:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 9:11:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 9:11:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 21:11:52.572118 28117 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.117.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.117.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/rangerkms_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.117.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/rangerkms_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.117.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.117.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.117.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/HTTP_127.27.117.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 21:11:52 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 21:11:52.657341 28117 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 21:11:52.666901 28117 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 21:11:53.409090 28117 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 21:11:53.409181 28117 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms
I20260810 21:11:53.409199 28117 mini_ranger_kms.cc:227] Using postgres at 127.27.117.84:37603
2026-08-10 21:11:53,584 [I] DB FLAVOR :POSTGRES
2026-08-10 21:11:53,585 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 21:11:53,585 [I] Checking connection
2026-08-10 21:11:53,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 21:11:53,856 [I] connection success
2026-08-10 21:11:53,856 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 21:11:53,856 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 21:11:53,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 21:11:54,108 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 21:11:54,108 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 21:11:54,108 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 21:11:54,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.117.84:37603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 21:11:54,377 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 21:11:54.863807 28117 mini_ranger_kms.cc:310] Created kms service
I20260810 21:11:55.063792 28117 mini_ranger_kms.cc:326] Created kudu user
I20260810 21:11:55.107764 28117 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 21:11:55 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:11:55.515362 28117 mini_ranger_kms.cc:384] Added ranger policy
I20260810 21:11:55.515487 28117 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksacWUR/build/release/bin/postgresql.jar:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksacWUR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 21:11:55.515518 28117 mini_ranger_kms.cc:240] Using host: 127.27.117.84
I20260810 21:11:55.519912 28117 mini_ranger_kms.cc:276] Ranger KMS PID: 17165
I20260810 21:11:55.520015 28117 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.117.84:44291
Aug 10, 2026 9:11:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 9:11:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 9:11:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 9:11:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 9:11:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 9:11:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 21:11:57 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786396317, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.117.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 21:11:58 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396317, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.117.84@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:11:58.418305 28117 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.761s user 0.005s sys 0.154s
I20260810 21:11:58.418429 28117 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 21:11:58.418475 28117 mini_ranger_kms.cc:401] 127.27.117.84:44291/kms/v1/keys
Aug 10 21:11:58 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396317, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.117.84@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
Aug 10 21:12:00 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:12:00.949656 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42051
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:42051
--ranger_config_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:01.033250 17255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:01.033444 17255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:01.033465 17255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:01.034616 17255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:01.034662 17255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:01.034675 17255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:01.034689 17255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:01.034701 17255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:01.036408 17255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.117.84:44291/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:42051
--ranger_config_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:01.036660 17255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:01.036926 17255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:01.041935 17266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:01.042017 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:01.042078 17255 server_base.cc:1034] running on GCE node
W20260810 21:12:01.041941 17265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:01.042450 17255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:01.042788 17255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:01.043949 17255 hybrid_clock.cc:648] HybridClock initialized: now 1786396321043925 us; error 46 us; skew 500 ppm
Aug 10 21:12:01 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:01.051118 17255 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:01.051591 17255 webserver.cc:460] Webserver started at http://127.27.117.126:44253/ using document root <none> and password file <none>
I20260810 21:12:01.051829 17255 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:01.051890 17255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:01.052034 17255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:12:01 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:01.099763 17255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "fc5553cccd6a4d8c82e5e85907d6143b"
format_stamp: "Formatted at 2026-08-10 21:12:01 on dist-test-slave-2kcd"
server_key: "053c217cd96f87bd68b0dca6e8087987"
server_key_iv: "357f8a6ec4dcd6f27a860c22bb452f46"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:01.100221 17255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "fc5553cccd6a4d8c82e5e85907d6143b"
format_stamp: "Formatted at 2026-08-10 21:12:01 on dist-test-slave-2kcd"
server_key: "053c217cd96f87bd68b0dca6e8087987"
server_key_iv: "357f8a6ec4dcd6f27a860c22bb452f46"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:01.102468 17255 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 21:12:01.115155 17281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:01.115389 17255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:12:01.115513 17255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "fc5553cccd6a4d8c82e5e85907d6143b"
format_stamp: "Formatted at 2026-08-10 21:12:01 on dist-test-slave-2kcd"
server_key: "053c217cd96f87bd68b0dca6e8087987"
server_key_iv: "357f8a6ec4dcd6f27a860c22bb452f46"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:01.115626 17255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:01.142123 17255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:01.144057 17255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:01.144233 17255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:01.149299 17255 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:42051
I20260810 21:12:01.149380 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:42051 every 8 connection(s)
I20260810 21:12:01.149755 17255 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:01.150512 17373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:01.153090 17373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Bootstrap starting.
I20260810 21:12:01.153743 17373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:01.154052 17373 log.cc:824] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:01.154800 17373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: No bootstrap required, opened a new log
I20260810 21:12:01.155395 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 17255
I20260810 21:12:01.155503 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:12:01.155682 28117 external_mini_cluster.cc:1428] Setting key 2f160b56f345ad97429af68cc22253ad
I20260810 21:12:01.157346 17373 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:01.157516 17373 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:01.157581 17373 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5553cccd6a4d8c82e5e85907d6143b, State: Initialized, Role: FOLLOWER
I20260810 21:12:01.157750 17373 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:01.157855 17373 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:01.157892 17373 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:01.157969 17373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:01.158730 17373 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:01.158856 17373 leader_election.cc:302] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5553cccd6a4d8c82e5e85907d6143b; no voters:
I20260810 21:12:01.159090 17373 leader_election.cc:288] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:12:01.159118 17384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:01.159375 17384 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 LEADER]: Becoming Leader. State: Replica: fc5553cccd6a4d8c82e5e85907d6143b, State: Running, Role: LEADER
I20260810 21:12:01.159471 17373 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:01.159564 17384 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:01.159941 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } } }
I20260810 21:12:01.160203 17387 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc5553cccd6a4d8c82e5e85907d6143b. Latest consensus state: current_term: 1 leader_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } } }
I20260810 21:12:01.160351 17387 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:01.160979 17373 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 21:12:01.161252 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:01.165436 17405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:01.166042 17405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 21:12:01 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:01.172645 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:01.156224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.27.117.126:42051)
0810 21:12:01.156398 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:01.156407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:01.156442 (+ 35us) server_negotiation.cc:407] Connection header received
0810 21:12:01.157358 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:01.157369 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:01.157437 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:01.157598 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:01.158576 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:01.160054 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:01.161208 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:01.164555 (+ 3347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:01.164734 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:01.168256 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:01.168260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:01.168260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:01.168272 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:01.170395 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:01.170819 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:01.170822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:01.170822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:01.170937 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:01.171151 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:01.171152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:01.171152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:01.171456 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:01.171595 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:01.172235 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 21:12:01.172407 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260810 21:12:01.173503 17405 catalog_manager.cc:1347] Generated new cluster ID: c6566fa23da4451a95bdda7bf017410b
I20260810 21:12:01.173552 17405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:12:01.180591 17405 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:12:01.181321 17405 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:01.190431 17405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Generated new TSK 0
I20260810 21:12:01.190632 17405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:12:01.901396 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42051
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:02.020913 17450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:02.021190 17450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:02.021227 17450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:02.023023 17450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:02.023092 17450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:02.023137 17450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:02.025542 17450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.117.84:44291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:02.025946 17450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:02.026283 17450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:02.032004 17461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.032169 17458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.032279 17459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.032640 17450 server_base.cc:1034] running on GCE node
I20260810 21:12:02.032883 17450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:02.033169 17450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:02.034374 17450 hybrid_clock.cc:648] HybridClock initialized: now 1786396322034359 us; error 45 us; skew 500 ppm
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:02.037611 17450 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:02.038177 17450 webserver.cc:460] Webserver started at http://127.27.117.65:34837/ using document root <none> and password file <none>
I20260810 21:12:02.038457 17450 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:02.038529 17450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:02.038698 17450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:02.052593 17450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "034a6547b5634953b1b34fb08b3363ca"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "e908e45eb091246b0466c5e99faa049a"
server_key_iv: "6fff00f9362e74012ea49888b071e6e6"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.053164 17450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "034a6547b5634953b1b34fb08b3363ca"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "e908e45eb091246b0466c5e99faa049a"
server_key_iv: "6fff00f9362e74012ea49888b071e6e6"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.054574 17450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:02.068898 17473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.069131 17450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 21:12:02.069235 17450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "034a6547b5634953b1b34fb08b3363ca"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "e908e45eb091246b0466c5e99faa049a"
server_key_iv: "6fff00f9362e74012ea49888b071e6e6"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.069339 17450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:02.084956 17450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:02.087210 17450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:02.087684 17450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:02.087988 17450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:02.088426 17450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:02.088536 17450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.088660 17450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:02.088747 17450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.096961 17450 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:35963
I20260810 21:12:02.097420 17450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:12:02.098273 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 17450
I20260810 21:12:02.098373 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:12:02.098552 28117 external_mini_cluster.cc:1428] Setting key c322ce749abb0e412e4cefc3b5802eb0
I20260810 21:12:02.105289 17673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:35963 every 8 connection(s)
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:02.124655 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:02.105868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42051 (local address 127.27.117.65:45203)
0810 21:12:02.112932 (+ 7064us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:02.112990 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:02.113597 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:02.115194 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:02.115199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:02.115419 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:02.115679 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:02.115685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.116721 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.116722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:02.117936 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:02.117939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.118443 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.118444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:02.118621 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:02.119204 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:02.119214 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:02.121000 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:02.123074 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:02.123081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:02.123083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:02.123344 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:02.123565 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:02.123566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:02.123566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:02.123686 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:02.123986 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:02.123993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:02.124139 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 21:12:02.124270 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 21:12:02.124422 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7020,"thread_start_us":76,"threads_started":1}
I20260810 21:12:02.125021 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:02.106147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:45203 (local address 127.27.117.126:42051)
0810 21:12:02.106320 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:02.106322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:02.114938 (+ 8616us) server_negotiation.cc:407] Connection header received
0810 21:12:02.114971 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:02.114972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:02.114986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:02.115061 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:02.115792 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.116605 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.118033 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.118328 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.118394 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:02.121116 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:02.121118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:02.121118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:02.121125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:02.122980 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:02.123426 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:02.123427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:02.123427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:02.123499 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:02.123761 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:02.123762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:02.123763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:02.123894 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:02.123957 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:02.124846 (+ 889us) server_negotiation.cc:299] Negotiation successful
0810 21:12:02.124931 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260810 21:12:02.125696 17675 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42051
I20260810 21:12:02.125787 17675 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
I20260810 21:12:02.125972 17675 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:02.126487 17305 ts_manager.cc:194] Registered new tserver with Master: 034a6547b5634953b1b34fb08b3363ca (127.27.117.65:35963)
I20260810 21:12:02.127269 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:45203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:12:02.145346 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42051
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:02.249775 17685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:02.250136 17685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:02.250247 17685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:02.252439 17685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:02.252552 17685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:02.252650 17685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:02.254963 17685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.117.84:44291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:02.255427 17685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:02.255757 17685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:12:02.262670 17685 server_base.cc:1034] running on GCE node
W20260810 21:12:02.262874 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.263011 17697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.263788 17695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.264046 17685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:02.264314 17685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:02.265470 17685 hybrid_clock.cc:648] HybridClock initialized: now 1786396322265453 us; error 35 us; skew 500 ppm
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:02.268342 17685 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:02.268911 17685 webserver.cc:460] Webserver started at http://127.27.117.66:38017/ using document root <none> and password file <none>
I20260810 21:12:02.269202 17685 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:02.269285 17685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:02.269441 17685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:02.287298 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "58f0ac5b7a2ca8c12f800ab9e6e7f1f9"
server_key_iv: "f62aa0357d68beedb2d64f4460c59a12"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.292958 17685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "58f0ac5b7a2ca8c12f800ab9e6e7f1f9"
server_key_iv: "f62aa0357d68beedb2d64f4460c59a12"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.297422 17685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 21:12:02.323150 17713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.323383 17685 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.000s
I20260810 21:12:02.323511 17685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "58f0ac5b7a2ca8c12f800ab9e6e7f1f9"
server_key_iv: "f62aa0357d68beedb2d64f4460c59a12"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.323621 17685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:02.338497 17685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:02.340596 17685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:02.340757 17685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:02.341076 17685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:02.341464 17685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:02.341518 17685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.341560 17685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:02.341586 17685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.349727 17685 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:34911
I20260810 21:12:02.350129 17685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:12:02.357195 17919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:34911 every 8 connection(s)
I20260810 21:12:02.357509 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 17685
I20260810 21:12:02.357612 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:12:02.357803 28117 external_mini_cluster.cc:1428] Setting key 72da8671500682eb05aa2093cccddbd3
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
I20260810 21:12:02.368641 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:02.357132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:42683 (local address 127.27.117.126:42051)
0810 21:12:02.357183 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:02.357185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:02.357583 (+ 398us) server_negotiation.cc:407] Connection header received
0810 21:12:02.358388 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:02.358390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:02.358403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:02.358436 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:02.359298 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.360050 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.361693 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.362025 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.362110 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:02.365079 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:02.365081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:02.365081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:02.365091 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:02.366934 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:02.367453 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:02.367454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:02.367455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:02.367516 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:02.367814 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:02.367815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:02.367815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:02.367979 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:02.368068 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:02.368386 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 21:12:02.368478 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:12:02.368791 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:02.357109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42051 (local address 127.27.117.66:42683)
0810 21:12:02.357437 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:02.357485 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:02.358263 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:02.358632 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:02.358636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:02.358907 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:02.359183 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:02.359189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.360190 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.360191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:02.361571 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:02.361574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.362218 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.362220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:02.362389 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:02.363150 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:02.363160 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:02.364956 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:02.367031 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:02.367038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:02.367040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:02.367369 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:02.367597 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:02.367598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:02.367598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:02.367731 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:02.368064 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:02.368070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:02.368229 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 21:12:02.368362 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 21:12:02.368514 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":95,"threads_started":1}
I20260810 21:12:02.369513 17920 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42051
I20260810 21:12:02.369619 17920 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:02.369839 17920 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:02.370249 17305 ts_manager.cc:194] Registered new tserver with Master: 4ebe8b2902ba47d8b1b79ee30d9b8992 (127.27.117.66:34911)
I20260810 21:12:02.370747 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:42683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:12:02.389664 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42051
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:02.526091 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:02.526518 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:02.526641 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:02.528682 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:02.528815 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:02.528934 17933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:02.531587 17933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.117.84:44291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:02.531884 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:02.532193 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:02.538998 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.539108 17938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.539362 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.540211 17933 server_base.cc:1034] running on GCE node
I20260810 21:12:02.540457 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:02.540732 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:02.541904 17933 hybrid_clock.cc:648] HybridClock initialized: now 1786396322541892 us; error 22 us; skew 500 ppm
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:02.544986 17933 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:02.545469 17933 webserver.cc:460] Webserver started at http://127.27.117.67:36221/ using document root <none> and password file <none>
I20260810 21:12:02.545751 17933 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:02.545805 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:02.545928 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:02.558724 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "7e31ccc450514c3a9c91307e7df4eda7"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "3db80bc1023d9c0a918073643d0bae84"
server_key_iv: "c57856282f0bd9e43233705ca507a7fc"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.559207 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "7e31ccc450514c3a9c91307e7df4eda7"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "3db80bc1023d9c0a918073643d0bae84"
server_key_iv: "c57856282f0bd9e43233705ca507a7fc"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.560999 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 21:12:02.578182 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.578465 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 21:12:02.578579 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "7e31ccc450514c3a9c91307e7df4eda7"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "3db80bc1023d9c0a918073643d0bae84"
server_key_iv: "c57856282f0bd9e43233705ca507a7fc"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.578702 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:02.591327 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:02.593990 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:02.594147 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:02.594491 17933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:02.594889 17933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:02.594935 17933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.594964 17933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:02.594981 17933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.609910 17933 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:46181
I20260810 21:12:02.610062 18168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:46181 every 8 connection(s)
I20260810 21:12:02.610468 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:12:02.619108 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 17933
I20260810 21:12:02.619240 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:12:02.619472 28117 external_mini_cluster.cc:1428] Setting key 179221eb2817b620bbaa594e172184ae
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:02.625098 17681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:02.611782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53109 (local address 127.27.117.126:42051)
0810 21:12:02.611832 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:02.611834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:02.611922 (+ 88us) server_negotiation.cc:407] Connection header received
0810 21:12:02.613062 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:02.613064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:02.613076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:02.613154 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:02.614369 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.615185 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.617503 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.617692 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.617753 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:02.621276 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:02.621277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:02.621277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:02.621287 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:02.623293 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:02.623722 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:02.623723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:02.623724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:02.623837 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:02.624123 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:02.624124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:02.624125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:02.624354 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:02.624506 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:02.624815 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 21:12:02.624960 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:12:02.625144 18174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:02.611452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42051 (local address 127.27.117.67:53109)
0810 21:12:02.611756 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:02.611814 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:02.612907 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:02.613265 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:02.613269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:02.613742 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:02.614229 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:02.614243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.615363 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.615364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:02.617356 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:02.617360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:02.617854 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:02.617856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:02.618072 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:02.618750 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:02.618769 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:02.621167 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:02.623394 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:02.623399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:02.623401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:02.623619 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:02.623924 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:02.623924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:02.623925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:02.624022 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:02.624496 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:02.624499 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:02.624608 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 21:12:02.624689 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:12:02.624863 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":167,"threads_started":1}
I20260810 21:12:02.625844 18172 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42051
I20260810 21:12:02.625957 18172 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:02.626139 18172 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:02.626613 17305 ts_manager.cc:194] Registered new tserver with Master: 7e31ccc450514c3a9c91307e7df4eda7 (127.27.117.67:46181)
I20260810 21:12:02.627281 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:53109
I20260810 21:12:02.631163 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:12:02.631767 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 17450
I20260810 21:12:02.645196 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 17685
I20260810 21:12:02.654075 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 17933
I20260810 21:12:02.667275 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 17255
I20260810 21:12:02.683625 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42051
--webserver_interface=127.27.117.126
--webserver_port=44253
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:42051
--ranger_config_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:02.764657 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:02.764951 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:02.764981 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:02.766081 18181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:02.766117 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:02.766131 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:02.766144 18181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:02.766155 18181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:02.767684 18181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:42051
--ranger_config_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:02.767872 18181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:02.768033 18181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:02.772830 18188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.772825 18187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.773051 18181 server_base.cc:1034] running on GCE node
W20260810 21:12:02.772805 18190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.773348 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:02.773672 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:02.774824 18181 hybrid_clock.cc:648] HybridClock initialized: now 1786396322774802 us; error 38 us; skew 500 ppm
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:02.777345 18181 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:02.777801 18181 webserver.cc:460] Webserver started at http://127.27.117.126:44253/ using document root <none> and password file <none>
I20260810 21:12:02.778043 18181 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:02.778093 18181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:02.825321 18181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:02.826314 18203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.826556 18181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:02.826666 18181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "fc5553cccd6a4d8c82e5e85907d6143b"
format_stamp: "Formatted at 2026-08-10 21:12:01 on dist-test-slave-2kcd"
server_key: "053c217cd96f87bd68b0dca6e8087987"
server_key_iv: "357f8a6ec4dcd6f27a860c22bb452f46"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.827147 18181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:02.843681 18181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:02.845304 18181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:02.845419 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:02.849673 18181 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:42051
I20260810 21:12:02.849722 18286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:42051 every 8 connection(s)
I20260810 21:12:02.850090 18181 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:02.851282 18287 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:12:02.851951 18287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Bootstrap starting.
I20260810 21:12:02.859807 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 18181
I20260810 21:12:02.860178 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:35963
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:42051
--webserver_port=34837
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:02.860478 18287 log.cc:824] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:02.862610 18287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:02.862936 18287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Bootstrap complete.
I20260810 21:12:02.865628 18287 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:02.865875 18287 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5553cccd6a4d8c82e5e85907d6143b, State: Initialized, Role: FOLLOWER
I20260810 21:12:02.866048 18287 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:02.866127 18287 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:02.866180 18287 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:02.866233 18287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 1 FOLLOWER]: Advancing to term 2
I20260810 21:12:02.867255 18287 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:02.867352 18287 leader_election.cc:302] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc5553cccd6a4d8c82e5e85907d6143b; no voters:
I20260810 21:12:02.867584 18287 leader_election.cc:288] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 21:12:02.867621 18291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 21:12:02.867744 18291 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [term 2 LEADER]: Becoming Leader. State: Replica: fc5553cccd6a4d8c82e5e85907d6143b, State: Running, Role: LEADER
I20260810 21:12:02.867873 18287 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:02.867847 18291 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } }
I20260810 21:12:02.868168 18293 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc5553cccd6a4d8c82e5e85907d6143b. Latest consensus state: current_term: 2 leader_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } } }
I20260810 21:12:02.868413 18293 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:02.868335 18292 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc5553cccd6a4d8c82e5e85907d6143b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42051 } } }
I20260810 21:12:02.868737 18292 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:02.868909 18310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:02.869436 18310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:02.869608 18310 catalog_manager.cc:1259] Loaded cluster ID: c6566fa23da4451a95bdda7bf017410b
I20260810 21:12:02.869657 18310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:12:02.869848 18287 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 21:12:02.873328 18310 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:02.873888 18310 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Loaded TSK: 0
I20260810 21:12:02.874169 18310 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 21:12:02.947753 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:02.948006 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:02.948043 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:02.949785 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:02.949869 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:02.949908 18290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:02.952005 18290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:35963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=34837
--tserver_master_addrs=127.27.117.126:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:02.952251 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:02.952524 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:02.959106 18353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:02.959074 18352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.959568 18290 server_base.cc:1034] running on GCE node
W20260810 21:12:02.959795 18357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:02.960060 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:02.960340 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:02.961519 18290 hybrid_clock.cc:648] HybridClock initialized: now 1786396322961484 us; error 48 us; skew 500 ppm
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:02.965195 18290 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:02.965685 18290 webserver.cc:460] Webserver started at http://127.27.117.65:34837/ using document root <none> and password file <none>
I20260810 21:12:02.965942 18290 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:02.966012 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 21:12:02 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:02.986846 18290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 21:12:02.989454 18376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:02.989638 18290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260810 21:12:02.989740 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "034a6547b5634953b1b34fb08b3363ca"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "e908e45eb091246b0466c5e99faa049a"
server_key_iv: "6fff00f9362e74012ea49888b071e6e6"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:02.990226 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:03.022718 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:03.025295 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:03.025452 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:03.025686 18290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:03.026111 18290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:03.026190 18290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.026237 18290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:03.026270 18290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.032779 18290 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:35963
I20260810 21:12:03.033030 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:35963 every 8 connection(s)
I20260810 21:12:03.033293 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:12:03.039629 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 18290
I20260810 21:12:03.039994 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:34911
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:42051
--webserver_port=38017
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:03.083226 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.034810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42051 (local address 127.27.117.65:44139)
0810 21:12:03.048957 (+ 14147us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:03.049019 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:03.049973 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:03.050829 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:03.050833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:03.051095 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:03.051384 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:03.051390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.053166 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.053167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:03.054652 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:03.054655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.063923 (+ 9268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.063925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:03.064210 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.065086 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:03.065096 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:03.067453 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:03.080899 (+ 13446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:03.080912 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:03.080914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:03.081293 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:03.081658 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:03.081659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:03.081659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:03.081825 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:03.082398 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:03.082405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:03.082582 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 21:12:03.082703 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 21:12:03.082911 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14094,"thread_start_us":83,"threads_started":1}
I20260810 21:12:03.083988 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.035229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:44139 (local address 127.27.117.126:42051)
0810 21:12:03.050262 (+ 15033us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:03.050271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:03.050306 (+ 35us) server_negotiation.cc:407] Connection header received
0810 21:12:03.050475 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:03.050481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:03.050538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:03.050675 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:03.051533 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.053046 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.062907 (+ 9861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.063601 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.063809 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.075254 (+ 11445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:03.075260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:03.075260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:03.075274 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:03.080740 (+ 5466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.081416 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.081420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.081421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.081569 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.081924 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.081925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.081926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.082274 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:03.082353 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:03.083596 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0810 21:12:03.083735 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14948,"thread_start_us":104,"threads_started":1}
I20260810 21:12:03.084908 18567 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42051
I20260810 21:12:03.085008 18567 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.085213 18567 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
W20260810 21:12:03.086094 18567 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:42051 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 21:12:03.086158 18567 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.086211 18567 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:03.086521 18567 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.086588 18567 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
W20260810 21:12:03.086871 18567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:12:03.189285 18572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:03.189512 18572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:03.189548 18572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:03.191159 18572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:03.191290 18572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:03.191395 18572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:03.193519 18572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:34911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=38017
--tserver_master_addrs=127.27.117.126:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:03.193960 18572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:03.194267 18572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:03.200520 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:03.200687 18572 server_base.cc:1034] running on GCE node
W20260810 21:12:03.200505 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:03.204560 18584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:03.204905 18572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:03.205152 18572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:03.206308 18572 hybrid_clock.cc:648] HybridClock initialized: now 1786396323206279 us; error 45 us; skew 500 ppm
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:03.209504 18572 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:03.209988 18572 webserver.cc:460] Webserver started at http://127.27.117.66:38017/ using document root <none> and password file <none>
I20260810 21:12:03.210253 18572 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:03.210319 18572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:03.234295 18572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:03.235436 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.235630 18572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:03.235733 18572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "58f0ac5b7a2ca8c12f800ab9e6e7f1f9"
server_key_iv: "f62aa0357d68beedb2d64f4460c59a12"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:03.236232 18572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:03.264979 18572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:03.266897 18572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:03.267045 18572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:03.267321 18572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:03.267637 18572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:03.267668 18572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.267709 18572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:03.267774 18572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.275151 18572 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:34911
I20260810 21:12:03.275274 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:34911 every 8 connection(s)
I20260810 21:12:03.275571 18572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:03.281862 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 18572
I20260810 21:12:03.282150 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.117.84:44291/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:46181
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:42051
--webserver_port=36221
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:03.284914 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.276283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:37513 (local address 127.27.117.126:42051)
0810 21:12:03.276329 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:03.276332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:03.276540 (+ 208us) server_negotiation.cc:407] Connection header received
0810 21:12:03.277197 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:03.277198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:03.277210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:03.277257 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:03.277868 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.278318 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.279504 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.279724 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.279800 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.282028 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:03.282030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:03.282030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:03.282037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:03.283351 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.283770 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.283771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.283771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.283831 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.284119 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.284120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.284121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.284249 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:03.284307 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:03.284662 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 21:12:03.284759 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:12:03.285553 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.276291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42051 (local address 127.27.117.66:37513)
0810 21:12:03.276430 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:03.276469 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:03.277093 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:03.277328 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:03.277332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:03.277491 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:03.277747 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:03.277752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.278426 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.278427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:03.279406 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:03.279409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.279840 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.279842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:03.279972 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.280464 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:03.280472 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:03.281922 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:03.283451 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:03.283458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:03.283459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:03.283683 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:03.283907 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:03.283908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:03.283909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:03.284042 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:03.284354 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:03.284360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:03.284511 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 21:12:03.285267 (+ 756us) client_negotiation.cc:240] Negotiation successful
0810 21:12:03.285398 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":84,"threads_started":1}
I20260810 21:12:03.286172 18822 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42051
I20260810 21:12:03.286253 18822 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.286425 18822 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
W20260810 21:12:03.287032 18822 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:42051 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 21:12:03.287158 18822 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.287262 18822 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:03.287632 18822 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.287755 18822 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
W20260810 21:12:03.288090 18822 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:12:03.385253 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:03.385464 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:03.385504 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:03.386616 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:03.386683 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:03.386727 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:03.388326 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.117.84:44291/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:46181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=36221
--tserver_master_addrs=127.27.117.126:42051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:03.388705 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:03.389008 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:03.394724 18835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:03.394833 18836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:03.394706 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:03.395432 18828 server_base.cc:1034] running on GCE node
I20260810 21:12:03.395613 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:03.395809 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:03.396930 18828 hybrid_clock.cc:648] HybridClock initialized: now 1786396323396907 us; error 39 us; skew 500 ppm
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:03.399490 18828 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:03.399973 18828 webserver.cc:460] Webserver started at http://127.27.117.67:36221/ using document root <none> and password file <none>
I20260810 21:12:03.400230 18828 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:03.400308 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:03.416137 18828 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 21:12:03.417248 18850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.417490 18828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:12:03.417590 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "7e31ccc450514c3a9c91307e7df4eda7"
format_stamp: "Formatted at 2026-08-10 21:12:02 on dist-test-slave-2kcd"
server_key: "3db80bc1023d9c0a918073643d0bae84"
server_key_iv: "c57856282f0bd9e43233705ca507a7fc"
server_key_version: "kuduclusterkey@0"
I20260810 21:12:03.418066 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:03.431757 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:03.434242 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:03.434453 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:03.434739 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:03.435149 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:03.435252 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.435318 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:03.435365 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:03.443153 18828 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:46181
I20260810 21:12:03.443305 19054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:46181 every 8 connection(s)
I20260810 21:12:03.443676 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:12:03.450044 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 18828
Aug 10 21:12:03 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:03.455706 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.444477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42051 (local address 127.27.117.67:56301)
0810 21:12:03.444932 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:03.444980 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:03.445663 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:03.445944 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:03.445948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:03.446202 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:03.446488 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:03.446494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.447501 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.447502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:03.448870 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:03.448873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.449317 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.449319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:03.449463 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.450369 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:03.450377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:03.452023 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:03.453931 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:03.453938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:03.453940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:03.454247 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:03.454481 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:03.454482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:03.454482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:03.454619 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:03.455026 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:03.455034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:03.455192 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 21:12:03.455309 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 21:12:03.455473 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":81,"threads_started":1}
I20260810 21:12:03.456511 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.444774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:56301 (local address 127.27.117.126:42051)
0810 21:12:03.444854 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:03.444857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:03.445068 (+ 211us) server_negotiation.cc:407] Connection header received
0810 21:12:03.445788 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:03.445790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:03.445809 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:03.445882 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:03.446611 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.447375 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.448976 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.449204 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.449291 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.452129 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:03.452130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:03.452130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:03.452138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:03.453816 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.454342 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.454343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.454343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.454406 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.454701 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.454702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.454702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.454922 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:03.454994 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:03.456317 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0810 21:12:03.456402 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":44}
I20260810 21:12:03.457075 19055 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42051
I20260810 21:12:03.457185 19055 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.457368 19055 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
W20260810 21:12:03.458086 19055 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:42051 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 21:12:03.458983 19055 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.459105 19055 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:03.459543 19055 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:03.459666 19055 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
W20260810 21:12:03.459970 19055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 21:12:03.460326 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:03.450742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.27.117.126:42051)
0810 21:12:03.451282 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:03.451284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:03.451290 (+ 6us) server_negotiation.cc:407] Connection header received
0810 21:12:03.451311 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:03.451312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:03.451323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:03.452582 (+ 1259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:03.452618 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.453420 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.454499 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:03.454868 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:03.454926 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:03.456310 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:03.456311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:03.456311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:03.456318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:03.458859 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.459324 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.459326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.459326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.459458 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:03.459811 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:03.459812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:03.459812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:03.459941 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:03.460106 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:03.460156 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 21:12:03.460240 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":519,"thread_start_us":76,"threads_started":1}
Aug 10 21:12:04 dist-test-slave-2kcd krb5kdc[16485](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786396324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:04.087057 18567 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:04.087227 18567 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:04.087630 18227 ts_manager.cc:194] Registered new tserver with Master: 034a6547b5634953b1b34fb08b3363ca (127.27.117.65:35963)
I20260810 21:12:04.088421 18227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:44139
I20260810 21:12:04.288314 18822 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:04.288470 18822 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:04.288941 18227 ts_manager.cc:194] Registered new tserver with Master: 4ebe8b2902ba47d8b1b79ee30d9b8992 (127.27.117.66:34911)
I20260810 21:12:04.289521 18227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:37513
Aug 10 21:12:04 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
I20260810 21:12:04.460225 19055 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:04.460373 19055 heartbeater.cc:511] Master 127.27.117.126:42051 requested a full tablet report, sending...
I20260810 21:12:04.460733 18227 ts_manager.cc:194] Registered new tserver with Master: 7e31ccc450514c3a9c91307e7df4eda7 (127.27.117.67:46181)
I20260810 21:12:04.461272 18227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:56301
I20260810 21:12:04.467367 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:12:04.484215 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.468504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.27.117.126:42051)
0810 21:12:04.475931 (+ 7427us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.475933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.476221 (+ 288us) server_negotiation.cc:407] Connection header received
0810 21:12:04.476245 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.476246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.476257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.476313 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:04.477135 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.477852 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.478935 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.479247 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.479305 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.480897 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:04.480898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:04.480899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:04.480914 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:04.482778 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:04.483166 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:04.483167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:04.483167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:04.483226 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:04.483405 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:04.483406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:04.483407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:04.483539 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:04.483627 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.484030 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.484110 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7393,"thread_start_us":85,"threads_started":1}
I20260810 21:12:04.486611 18227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:12:04.489084 18227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:12:04.503129 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.496528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:34911 (local address 127.0.0.1:46258)
0810 21:12:04.497779 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:04.497789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:04.497837 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:04.498223 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:04.498227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:04.498229 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:04.498294 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.498295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.499926 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.499927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.501590 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.501593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.502815 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.502816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.502898 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.502924 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 21:12:04.502993 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 21:12:04.503041 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":119,"threads_started":1}
I20260810 21:12:04.503628 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.496626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.27.117.66:34911)
0810 21:12:04.497694 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.497700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.497896 (+ 196us) server_negotiation.cc:407] Connection header received
0810 21:12:04.497944 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.497946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.497963 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.498031 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:04.498386 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.499815 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.501683 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.502643 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.502718 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.502777 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.503475 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.503517 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1031,"thread_start_us":56,"threads_started":1}
I20260810 21:12:04.504649 18699 tablet_service.cc:1467] Processing CreateTablet for tablet ebd497511de940a1b84a5d027fd45c69 (DEFAULT_TABLE table=test-table [id=79ebc90c17a44118a877747b563ff70d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:04.505095 18699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd497511de940a1b84a5d027fd45c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:04.510792 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.496796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46181 (local address 127.0.0.1:45378)
0810 21:12:04.504278 (+ 7482us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:04.504287 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:04.504317 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:04.504882 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:04.504883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:04.504885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:04.504923 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.504924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.506197 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.506199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.507309 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.507312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.510513 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.510514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.510573 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.510576 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:04.510614 (+ 38us) client_negotiation.cc:240] Negotiation successful
0810 21:12:04.510719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7472,"thread_start_us":117,"threads_started":1}
I20260810 21:12:04.513811 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.496882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.27.117.67:46181)
0810 21:12:04.504455 (+ 7573us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.504460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.504469 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:04.504527 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.504529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.504544 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.504623 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:04.505011 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.506099 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.509380 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.510317 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.510423 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.510479 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.513691 (+ 3212us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.513735 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7535,"thread_start_us":44,"threads_started":1}
I20260810 21:12:04.514175 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.496241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:35963 (local address 127.0.0.1:35078)
0810 21:12:04.497349 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:04.497392 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:04.497466 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:04.507866 (+ 10400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:04.507867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:04.507869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:04.507980 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.507981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.510999 (+ 3018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.511000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.512155 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.512158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.513963 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.513964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.514039 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.514042 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:04.514073 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 21:12:04.514108 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":119,"threads_started":1}
I20260810 21:12:04.514540 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.496360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.27.117.65:35963)
0810 21:12:04.507703 (+ 11343us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.507708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.507717 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:04.507750 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.507752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.507768 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.508066 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:04.508181 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.509255 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.512598 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.513563 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.514337 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.514402 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.514427 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.514464 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11294,"thread_start_us":59,"threads_started":1}
I20260810 21:12:04.514508 18933 tablet_service.cc:1467] Processing CreateTablet for tablet ebd497511de940a1b84a5d027fd45c69 (DEFAULT_TABLE table=test-table [id=79ebc90c17a44118a877747b563ff70d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:04.514914 18933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd497511de940a1b84a5d027fd45c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:04.519301 18463 tablet_service.cc:1467] Processing CreateTablet for tablet ebd497511de940a1b84a5d027fd45c69 (DEFAULT_TABLE table=test-table [id=79ebc90c17a44118a877747b563ff70d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:04.519632 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebd497511de940a1b84a5d027fd45c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:04.523267 19111 tablet_bootstrap.cc:492] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: Bootstrap starting.
I20260810 21:12:04.523861 19110 tablet_bootstrap.cc:492] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Bootstrap starting.
I20260810 21:12:04.524444 19111 tablet_bootstrap.cc:654] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:04.524588 19110 tablet_bootstrap.cc:654] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:04.524663 19109 tablet_bootstrap.cc:492] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Bootstrap starting.
I20260810 21:12:04.525478 19110 log.cc:824] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:04.526132 19109 tablet_bootstrap.cc:654] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:04.526840 19110 tablet_bootstrap.cc:492] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: No bootstrap required, opened a new log
I20260810 21:12:04.526925 19110 ts_tablet_manager.cc:1397] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 21:12:04.526932 19109 log.cc:824] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:04.527784 19111 log.cc:824] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:04.528973 19109 tablet_bootstrap.cc:492] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: No bootstrap required, opened a new log
I20260810 21:12:04.529042 19109 ts_tablet_manager.cc:1397] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 21:12:04.531651 19110 raft_consensus.cc:348] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.531944 19110 raft_consensus.cc:374] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:04.532060 19110 raft_consensus.cc:729] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e31ccc450514c3a9c91307e7df4eda7, State: Initialized, Role: FOLLOWER
I20260810 21:12:04.532375 19110 consensus_queue.cc:260] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.532889 19110 ts_tablet_manager.cc:1428] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260810 21:12:04.533107 19055 heartbeater.cc:503] Master 127.27.117.126:42051 was elected leader, sending a full tablet report...
I20260810 21:12:04.533438 19109 raft_consensus.cc:348] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.533591 19109 raft_consensus.cc:374] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:04.533623 19109 raft_consensus.cc:729] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebe8b2902ba47d8b1b79ee30d9b8992, State: Initialized, Role: FOLLOWER
I20260810 21:12:04.533725 19109 consensus_queue.cc:260] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.534006 19109 ts_tablet_manager.cc:1428] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260810 21:12:04.534135 18822 heartbeater.cc:503] Master 127.27.117.126:42051 was elected leader, sending a full tablet report...
I20260810 21:12:04.540500 19111 tablet_bootstrap.cc:492] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: No bootstrap required, opened a new log
I20260810 21:12:04.540589 19111 ts_tablet_manager.cc:1397] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20260810 21:12:04.544265 19111 raft_consensus.cc:348] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.544490 19111 raft_consensus.cc:374] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:04.544575 19111 raft_consensus.cc:729] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034a6547b5634953b1b34fb08b3363ca, State: Initialized, Role: FOLLOWER
I20260810 21:12:04.544800 19111 consensus_queue.cc:260] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.545164 19111 ts_tablet_manager.cc:1428] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 21:12:04.545320 18567 heartbeater.cc:503] Master 127.27.117.126:42051 was elected leader, sending a full tablet report...
W20260810 21:12:04.695053 19058 tablet.cc:2369] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:12:04.706991 19121 raft_consensus.cc:482] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:04.707226 19121 raft_consensus.cc:504] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.707818 19121 leader_election.cc:288] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ebe8b2902ba47d8b1b79ee30d9b8992 (127.27.117.66:34911), 7e31ccc450514c3a9c91307e7df4eda7 (127.27.117.67:46181)
I20260810 21:12:04.713478 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.708542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:50153 (local address 127.27.117.67:46181)
0810 21:12:04.708631 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.708633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.709305 (+ 672us) server_negotiation.cc:407] Connection header received
0810 21:12:04.709443 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.709444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.709456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.709494 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:04.709893 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.710624 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.712222 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.712916 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.712975 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.712986 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.713344 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.713406 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:12:04.714061 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.708168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46181 (local address 127.27.117.65:50153)
0810 21:12:04.709217 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:04.709231 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:04.709342 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:04.709577 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:04.709577 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:04.709580 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:04.709769 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.709772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.710764 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.710765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.712079 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.712082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.713130 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.713132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.713230 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.713235 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:12:04.713770 (+ 535us) client_negotiation.cc:240] Negotiation successful
0810 21:12:04.713831 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":172,"threads_started":1}
I20260810 21:12:04.714756 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd497511de940a1b84a5d027fd45c69" candidate_uuid: "034a6547b5634953b1b34fb08b3363ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e31ccc450514c3a9c91307e7df4eda7" is_pre_election: true
I20260810 21:12:04.714960 18972 raft_consensus.cc:2454] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034a6547b5634953b1b34fb08b3363ca in term 0.
I20260810 21:12:04.715366 18379 leader_election.cc:302] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034a6547b5634953b1b34fb08b3363ca, 7e31ccc450514c3a9c91307e7df4eda7; no voters:
I20260810 21:12:04.715518 19121 raft_consensus.cc:2781] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:12:04.715583 19121 raft_consensus.cc:482] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:12:04.715627 19121 raft_consensus.cc:3037] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:04.717192 19121 raft_consensus.cc:504] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.717348 19121 leader_election.cc:288] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [CANDIDATE]: Term 1 election: Requested vote from peers 4ebe8b2902ba47d8b1b79ee30d9b8992 (127.27.117.66:34911), 7e31ccc450514c3a9c91307e7df4eda7 (127.27.117.67:46181)
I20260810 21:12:04.717588 18972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd497511de940a1b84a5d027fd45c69" candidate_uuid: "034a6547b5634953b1b34fb08b3363ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e31ccc450514c3a9c91307e7df4eda7"
I20260810 21:12:04.717671 18972 raft_consensus.cc:3037] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:04.718820 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.709012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:54829 (local address 127.27.117.66:34911)
0810 21:12:04.709075 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.709077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.710482 (+ 1405us) server_negotiation.cc:407] Connection header received
0810 21:12:04.710581 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.710582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.710592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.710619 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:04.710949 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.711657 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.717772 (+ 6115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.718370 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.718420 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.718427 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.718679 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.718729 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 21:12:04.718966 18972 raft_consensus.cc:2454] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034a6547b5634953b1b34fb08b3363ca in term 1.
I20260810 21:12:04.719192 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.708646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:34911 (local address 127.27.117.65:54829)
0810 21:12:04.710407 (+ 1761us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:04.710419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:04.710510 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:04.710683 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:04.710684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:04.710686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:04.710859 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.710861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.716273 (+ 5412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.716274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.717666 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:04.717670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.718486 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.718486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:04.718596 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.718599 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:04.719033 (+ 434us) client_negotiation.cc:240] Negotiation successful
0810 21:12:04.719078 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":175,"threads_started":1}
I20260810 21:12:04.719764 18730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd497511de940a1b84a5d027fd45c69" candidate_uuid: "034a6547b5634953b1b34fb08b3363ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" is_pre_election: true
I20260810 21:12:04.719915 18730 raft_consensus.cc:2454] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034a6547b5634953b1b34fb08b3363ca in term 0.
I20260810 21:12:04.720086 18729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebd497511de940a1b84a5d027fd45c69" candidate_uuid: "034a6547b5634953b1b34fb08b3363ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992"
I20260810 21:12:04.720167 18729 raft_consensus.cc:3037] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:04.721460 18729 raft_consensus.cc:2454] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034a6547b5634953b1b34fb08b3363ca in term 1.
I20260810 21:12:04.721750 18377 leader_election.cc:302] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034a6547b5634953b1b34fb08b3363ca, 4ebe8b2902ba47d8b1b79ee30d9b8992; no voters:
I20260810 21:12:04.722071 19121 raft_consensus.cc:2781] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:04.722229 19121 raft_consensus.cc:686] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 1 LEADER]: Becoming Leader. State: Replica: 034a6547b5634953b1b34fb08b3363ca, State: Running, Role: LEADER
I20260810 21:12:04.722354 19121 consensus_queue.cc:237] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } }
I20260810 21:12:04.723376 18227 catalog_manager.cc:5495] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 034a6547b5634953b1b34fb08b3363ca (127.27.117.65). New cstate: current_term: 1 leader_uuid: "034a6547b5634953b1b34fb08b3363ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034a6547b5634953b1b34fb08b3363ca" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 } health_report { overall_health: UNKNOWN } } }
Aug 10 21:12:04 dist-test-slave-2kcd krb5kdc[16485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for HTTP/127.27.117.84@KRBTEST.COM
W20260810 21:12:04.779462 18823 tablet.cc:2369] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:12:04.788635 18568 tablet.cc:2369] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:12:04.791258 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.786042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.27.117.65:35963)
0810 21:12:04.786177 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:04.786180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:04.786189 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:04.786236 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:04.786237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:04.786261 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:04.786339 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:04.786533 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.787285 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.790141 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:04.790460 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:04.790544 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:04.790772 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:04.790910 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:04.790987 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:04.791156 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 21:12:04.791181 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 21:12:04.793671 18972 raft_consensus.cc:1261] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 1 FOLLOWER]: Refusing update from remote peer 034a6547b5634953b1b34fb08b3363ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:04.793672 18729 raft_consensus.cc:1261] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 1 FOLLOWER]: Refusing update from remote peer 034a6547b5634953b1b34fb08b3363ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:04.794121 19126 consensus_queue.cc:1035] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e31ccc450514c3a9c91307e7df4eda7" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:04.794369 19126 consensus_queue.cc:1035] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebe8b2902ba47d8b1b79ee30d9b8992" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 34911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:04.800450 18227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:04.800562 18227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:04.801534 18227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc5553cccd6a4d8c82e5e85907d6143b: Sending DeleteTablet for 3 replicas of tablet ebd497511de940a1b84a5d027fd45c69
I20260810 21:12:04.801997 18933 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd497511de940a1b84a5d027fd45c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:04 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:45378
I20260810 21:12:04.802110 18463 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd497511de940a1b84a5d027fd45c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:04 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:35078
I20260810 21:12:04.802639 18699 tablet_service.cc:1514] Processing DeleteTablet for tablet ebd497511de940a1b84a5d027fd45c69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:04 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:46258
I20260810 21:12:04.802774 19146 tablet_replica.cc:330] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca: stopping tablet replica
I20260810 21:12:04.802917 19146 raft_consensus.cc:2229] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:12:04.803072 19146 raft_consensus.cc:2258] T ebd497511de940a1b84a5d027fd45c69 P 034a6547b5634953b1b34fb08b3363ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:04.803265 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 18290
I20260810 21:12:04.803429 19151 tablet_replica.cc:330] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: stopping tablet replica
I20260810 21:12:04.803694 19151 raft_consensus.cc:2229] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:04.803871 19151 raft_consensus.cc:2258] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:04.807135 19139 mvcc.cc:205] Tried to move back new op lower bound from 7317079346351894528 to 7317079346063409152. Current Snapshot: MvccSnapshot[applied={T|T < 7317079346351894528}]
I20260810 21:12:04.813138 19151 ts_tablet_manager.cc:1905] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:04.815318 19151 ts_tablet_manager.cc:1918] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:04.815393 19151 log.cc:1196] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/ebd497511de940a1b84a5d027fd45c69
I20260810 21:12:04.815683 19151 ts_tablet_manager.cc:1939] T ebd497511de940a1b84a5d027fd45c69 P 4ebe8b2902ba47d8b1b79ee30d9b8992: Deleting consensus metadata
I20260810 21:12:04.816164 18205 catalog_manager.cc:4841] TS 4ebe8b2902ba47d8b1b79ee30d9b8992 (127.27.117.66:34911): tablet ebd497511de940a1b84a5d027fd45c69 (table test-table [id=79ebc90c17a44118a877747b563ff70d]) successfully deleted
W20260810 21:12:04.816866 18208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:35963 (error 108)
I20260810 21:12:04.816985 19144 tablet_replica.cc:330] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: stopping tablet replica
I20260810 21:12:04.817099 19144 raft_consensus.cc:2229] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:04.817204 19144 raft_consensus.cc:2258] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:04.817279 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 18572
I20260810 21:12:04.817447 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:04.817305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:35963 (local address 127.0.0.1:35100)
0810 21:12:04.817368 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:04.817407 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 21:12:04.819938 18208 catalog_manager.cc:4584] TS 034a6547b5634953b1b34fb08b3363ca (127.27.117.65:35963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebd497511de940a1b84a5d027fd45c69: Network error: Client connection negotiation failed: client connection to 127.27.117.65:35963: connect: Connection refused (error 111)
I20260810 21:12:04.829473 19144 ts_tablet_manager.cc:1905] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:04.831713 19144 ts_tablet_manager.cc:1918] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:04.831815 19144 log.cc:1196] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/ebd497511de940a1b84a5d027fd45c69
I20260810 21:12:04.832134 19144 ts_tablet_manager.cc:1939] T ebd497511de940a1b84a5d027fd45c69 P 7e31ccc450514c3a9c91307e7df4eda7: Deleting consensus metadata
I20260810 21:12:04.832677 18207 catalog_manager.cc:4841] TS 7e31ccc450514c3a9c91307e7df4eda7 (127.27.117.67:46181): tablet ebd497511de940a1b84a5d027fd45c69 (table test-table [id=79ebc90c17a44118a877747b563ff70d]) successfully deleted
I20260810 21:12:04.833236 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 18828
I20260810 21:12:04.851315 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 18181
I20260810 21:12:04.869728 28117 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 21:12:05.217744 28117 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 21:12:05.217938 28117 mini_ranger.cc:66] Stopping Ranger...
I20260810 21:12:05.565356 28117 mini_ranger.cc:68] Stopped Ranger
2026-08-10 21:12:05.565 UTC [16538] LOG: received smart shutdown request
2026-08-10 21:12:05.570 UTC [16538] LOG: background worker "logical replication launcher" (PID 16555) exited with exit code 1
2026-08-10 21:12:05.570 UTC [16545] LOG: shutting down
2026-08-10 21:12:05.730 UTC [16538] LOG: database system is shut down
2026-08-10T21:12:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48593 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:12:05 dist-test-slave-2kcd krb5kdc[19173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:12:05 dist-test-slave-2kcd krb5kdc[19173](info): set up 2 sockets
Aug 10 21:12:05 dist-test-slave-2kcd krb5kdc[19173](info): commencing operation
krb5kdc: starting...
W20260810 21:12:07.912992 28117 mini_kdc.cc:121] Time spent starting KDC: real 2.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:12:07 dist-test-slave-2kcd krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:12:08.032474 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41825
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41825,127.27.117.125:45039,127.27.117.124:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:08.114638 19202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:08.114848 19202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:08.114873 19202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:08.115923 19202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:08.115960 19202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:08.115976 19202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:08.115991 19202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:08.116005 19202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:08.117439 19202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41825,127.27.117.125:45039,127.27.117.124:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:08.117630 19202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:08.117811 19202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:08.122676 19212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:08.122779 19210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:08.122850 19202 server_base.cc:1034] running on GCE node
W20260810 21:12:08.122658 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:08.123169 19202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:08.123454 19202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:08.124624 19202 hybrid_clock.cc:648] HybridClock initialized: now 1786396328124614 us; error 45 us; skew 500 ppm
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:08.131045 19202 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:08.131534 19202 webserver.cc:460] Webserver started at http://127.27.117.126:39279/ using document root <none> and password file <none>
I20260810 21:12:08.131783 19202 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:08.131853 19202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:08.132009 19202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:08.133121 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "4602737d094e4d91873eaac3df326e99"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "81874971bb86e1a5f8648a96caaf81ea"
server_key_iv: "95583bd9050f1a74f3119d3aa769128a"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.133478 19202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "4602737d094e4d91873eaac3df326e99"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "81874971bb86e1a5f8648a96caaf81ea"
server_key_iv: "95583bd9050f1a74f3119d3aa769128a"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.134819 19202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:12:08.135550 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:08.135717 19202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:08.135823 19202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "4602737d094e4d91873eaac3df326e99"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "81874971bb86e1a5f8648a96caaf81ea"
server_key_iv: "95583bd9050f1a74f3119d3aa769128a"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.135896 19202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:08.142722 19202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:08.144448 19202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:08.144614 19202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:08.148955 19202 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:41825
I20260810 21:12:08.149020 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:41825 every 8 connection(s)
I20260810 21:12:08.149376 19202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:08.150219 19319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:08.151778 19319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:08.157526 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 19202
I20260810 21:12:08.157622 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:12:08.157795 28117 external_mini_cluster.cc:1428] Setting key abad635b91accb8fd24ea0bce085abc0
I20260810 21:12:08.161166 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.152425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41825 (local address 127.0.0.1:45654)
0810 21:12:08.152584 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.152616 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.153184 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.153439 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.153441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.153693 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.153839 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.153842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.154767 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.154768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.155413 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.155414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.155873 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.155874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.155929 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.156371 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.156379 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.157980 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.159680 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.159688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.159691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.159926 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.160206 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.160207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.160207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.160287 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.160712 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.160716 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.160767 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.160888 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.161010 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260810 21:12:08.161270 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.152578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.27.117.126:41825)
0810 21:12:08.152707 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.152712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.152748 (+ 36us) server_negotiation.cc:407] Connection header received
0810 21:12:08.153319 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.153321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.153337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.153412 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.153917 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.154688 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.155482 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.155798 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.155881 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.158067 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.158074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.158075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.158081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.159575 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.160017 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.160020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.160021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.160133 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.160356 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.160357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.160357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.160641 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.160731 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.160951 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.161094 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":95,"thread_start_us":89,"threads_started":1}
I20260810 21:12:08.161940 19319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:12:08.162299 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.162182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60026)
0810 21:12:08.162229 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.162272 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 21:12:08.162443 19230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
I20260810 21:12:08.162871 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.162783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60038)
0810 21:12:08.162826 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.162850 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 21:12:08.162974 19319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:45039: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.117.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab.
Entry for principal kudu/127.27.117.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab.
I20260810 21:12:08.183450 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:45039
--webserver_interface=127.27.117.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab
--principal=kudu/127.27.117.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41825,127.27.117.125:45039,127.27.117.124:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:08.212205 19319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } attempt: 1
I20260810 21:12:08.212826 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.212633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60042)
0810 21:12:08.212706 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.212754 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 21:12:08.213217 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.213115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60048)
0810 21:12:08.213163 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.213195 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 21:12:08.213316 19319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:45039: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
W20260810 21:12:08.276273 19336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:08.276647 19336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:08.276722 19336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:08.277971 19336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:08.278033 19336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:08.278057 19336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:08.278086 19336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:08.278111 19336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:08.280247 19336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41825,127.27.117.125:45039,127.27.117.124:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab
--principal=kudu/127.27.117.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.117.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:08.280540 19336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:08.280851 19336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:12:08.281450 19319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } attempt: 2
I20260810 21:12:08.281944 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.281810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60060)
0810 21:12:08.281862 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.281909 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 21:12:08.282343 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.282244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60070)
0810 21:12:08.282276 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.282309 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260810 21:12:08.282449 19319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:45039: Network error: Client connection negotiation failed: client connection to 127.27.117.125:45039: connect: Connection refused (error 111)
W20260810 21:12:08.286749 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:08.286724 19348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:08.286695 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:08.286700 19336 server_base.cc:1034] running on GCE node
I20260810 21:12:08.287184 19336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:08.287572 19336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:08.288722 19336 hybrid_clock.cc:648] HybridClock initialized: now 1786396328288701 us; error 34 us; skew 500 ppm
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:08.291582 19336 init.cc:377] Logged in from keytab as kudu/127.27.117.125@KRBTEST.COM (short username kudu)
I20260810 21:12:08.292073 19336 webserver.cc:460] Webserver started at http://127.27.117.125:37417/ using document root <none> and password file <none>
I20260810 21:12:08.292325 19336 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:08.292371 19336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:08.292536 19336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:08.293751 19336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data/instance:
uuid: "74b65a4a58674e76ab65752723693750"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "7da170c3215473199b9548e50ef6713a"
server_key_iv: "08c5f53f6356b82b6d56f78726cd7599"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.294190 19336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal/instance:
uuid: "74b65a4a58674e76ab65752723693750"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "7da170c3215473199b9548e50ef6713a"
server_key_iv: "08c5f53f6356b82b6d56f78726cd7599"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.295538 19336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:12:08.296264 19359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:08.296437 19336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:12:08.296555 19336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal
uuid: "74b65a4a58674e76ab65752723693750"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "7da170c3215473199b9548e50ef6713a"
server_key_iv: "08c5f53f6356b82b6d56f78726cd7599"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.296631 19336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:08.306654 19336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:08.308370 19336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:08.308537 19336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:08.312721 19336 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.125:45039
I20260810 21:12:08.312753 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.125:45039 every 8 connection(s)
I20260810 21:12:08.313171 19336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
I20260810 21:12:08.314007 19445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:08.315467 19445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:12:08.318495 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 19336
I20260810 21:12:08.318583 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-1/wal/instance
I20260810 21:12:08.318764 28117 external_mini_cluster.cc:1428] Setting key 578b5ae90b7e5933b1bf62cf24dc5b10
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:08.325380 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.316127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41825 (local address 127.0.0.1:45658)
0810 21:12:08.316279 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.316321 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.316849 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.317045 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.317048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.317284 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.317470 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.317474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.318108 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.318109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.319211 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.319213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.319727 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.319728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.319833 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.320333 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.320340 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.322079 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.323981 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.323987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.323989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.324237 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.324460 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.324461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.324461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.324564 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.324911 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.324917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.325026 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.325114 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.325229 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260810 21:12:08.325402 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.316170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.27.117.126:41825)
0810 21:12:08.316214 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.316216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.316416 (+ 200us) server_negotiation.cc:407] Connection header received
0810 21:12:08.316926 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.316927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.316939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.316995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.317555 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.318027 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.319309 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.319637 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.319716 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.322183 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.322185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.322185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.322192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.323888 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.324317 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.324319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.324319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.324388 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.324634 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.324635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.324635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.324839 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.324918 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.325168 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.325266 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:12:08.326064 19445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.117.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.124@KRBTEST.COM" created.
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:12:08.335022 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.326412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.27.117.125:45039)
0810 21:12:08.326579 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.326584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.326609 (+ 25us) server_negotiation.cc:407] Connection header received
0810 21:12:08.326633 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.326635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.326650 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.326732 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.326988 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.328130 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.328875 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.329170 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.329267 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.331905 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.331913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.331913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.331920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.333727 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.334161 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.334164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.334164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.334224 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.334435 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.334436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.334436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.334569 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.334631 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.334790 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.334928 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260810 21:12:08.335073 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.326327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60086)
0810 21:12:08.326389 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.326402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.326441 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.326753 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.326753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.326864 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.326909 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.326910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.328245 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.328247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.328749 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.328751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.329287 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.329288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.329367 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.329785 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.329789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.331760 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.333824 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.333825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.333825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.334079 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.334295 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.334296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.334296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.334371 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.334637 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.334638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.334688 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.334769 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.334878 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 21:12:08.335736 19445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:12:08.336117 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.335973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48936)
0810 21:12:08.336022 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.336084 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 21:12:08.336262 19366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
I20260810 21:12:08.336731 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.336610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48940)
0810 21:12:08.336666 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.336699 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 21:12:08.336867 19445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:42249: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab.
Entry for principal kudu/127.27.117.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab.
I20260810 21:12:08.347235 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:42249
--webserver_interface=127.27.117.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab
--principal=kudu/127.27.117.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41825,127.27.117.125:45039,127.27.117.124:42249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:08.373626 19319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } attempt: 3
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:12:08.380812 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.374028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60096)
0810 21:12:08.374090 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.374105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.374181 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.374328 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.374329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.374519 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.374588 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.374591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.375293 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.375295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.375818 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.375820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.376207 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.376208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.376297 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.376856 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.376860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.378214 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.379739 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.379740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.379741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.379921 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.380126 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.380127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.380127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.380195 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.380450 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.380451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.380499 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.380587 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.380689 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 21:12:08.380811 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.374110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.27.117.125:45039)
0810 21:12:08.374196 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.374198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.374209 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:12:08.374240 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.374241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.374252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.374322 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.374677 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.375199 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.375924 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.376134 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.376210 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.378303 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.378305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.378305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.378313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.379651 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.379997 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.379999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.379999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.380057 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.380261 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.380262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.380262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.380374 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.380459 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.380613 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.380690 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 21:12:08.381479 19319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:12:08.381865 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.381752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48944)
0810 21:12:08.381802 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.381830 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 21:12:08.382216 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.382147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48946)
0810 21:12:08.382180 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.382203 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260810 21:12:08.382319 19319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:42249: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
I20260810 21:12:08.386101 19445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } attempt: 1
I20260810 21:12:08.386497 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.386372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48958)
0810 21:12:08.386436 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.386471 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 21:12:08.386852 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.386754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48972)
0810 21:12:08.386803 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.386831 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 21:12:08.387100 19445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:42249: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
I20260810 21:12:08.413545 19319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } attempt: 1
I20260810 21:12:08.414059 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.413919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48982)
0810 21:12:08.413990 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.414030 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 21:12:08.414486 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.414312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48984)
0810 21:12:08.414400 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.414448 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 21:12:08.414662 19319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:42249: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
W20260810 21:12:08.432644 19456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:08.432905 19456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:08.432932 19456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:08.434000 19456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:08.434034 19456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:08.434048 19456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:08.434060 19456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:08.434072 19456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:08.435493 19456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41825,127.27.117.125:45039,127.27.117.124:42249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab
--principal=kudu/127.27.117.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:42249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.117.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:08.435683 19456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:08.435849 19456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:08.440611 19465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:08.440625 19469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:08.440809 19456 server_base.cc:1034] running on GCE node
W20260810 21:12:08.440609 19463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:08.441147 19456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:08.441453 19456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:08.442632 19456 hybrid_clock.cc:648] HybridClock initialized: now 1786396328442615 us; error 29 us; skew 500 ppm
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:08.445051 19456 init.cc:377] Logged in from keytab as kudu/127.27.117.124@KRBTEST.COM (short username kudu)
I20260810 21:12:08.445474 19456 webserver.cc:460] Webserver started at http://127.27.117.124:42617/ using document root <none> and password file <none>
I20260810 21:12:08.445708 19456 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:08.445777 19456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:08.445932 19456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:08.446981 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data/instance:
uuid: "19d481877b934056afa63afb1ac08833"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "ec724564ae7fd6e46eb8da3ea139258b"
server_key_iv: "844068cf79d9be6f2e0f19ebed985b6e"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.447365 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal/instance:
uuid: "19d481877b934056afa63afb1ac08833"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "ec724564ae7fd6e46eb8da3ea139258b"
server_key_iv: "844068cf79d9be6f2e0f19ebed985b6e"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.448745 19456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:12:08.449527 19483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:08.449703 19456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:08.449810 19456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal
uuid: "19d481877b934056afa63afb1ac08833"
format_stamp: "Formatted at 2026-08-10 21:12:08 on dist-test-slave-2kcd"
server_key: "ec724564ae7fd6e46eb8da3ea139258b"
server_key_iv: "844068cf79d9be6f2e0f19ebed985b6e"
server_key_version: "encryptionkey@0"
I20260810 21:12:08.449895 19456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:08.455413 19445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } attempt: 2
I20260810 21:12:08.455838 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.455712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48986)
0810 21:12:08.455773 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.455812 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 21:12:08.456185 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.456083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48988)
0810 21:12:08.456129 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.456154 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 21:12:08.456269 19445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:42249: Network error: Client connection negotiation failed: client connection to 127.27.117.124:42249: connect: Connection refused (error 111)
I20260810 21:12:08.468194 19456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:08.470188 19456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:08.470371 19456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:08.474526 19456 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.124:42249
I20260810 21:12:08.474586 19583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.124:42249 every 8 connection(s)
I20260810 21:12:08.474961 19456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
I20260810 21:12:08.475898 19584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:08.477588 19584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:12:08.482231 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 19456
I20260810 21:12:08.482343 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786396166878326-28117-0/minicluster-data/master-2/wal/instance
I20260810 21:12:08.482514 28117 external_mini_cluster.cc:1428] Setting key c6586f4e8455fcce4492f0148b130fa1
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:08.486817 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.478332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.27.117.126:41825)
0810 21:12:08.478399 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.478400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.478621 (+ 221us) server_negotiation.cc:407] Connection header received
0810 21:12:08.479052 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.479053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.479062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.479130 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.479722 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.480209 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.481241 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.481417 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.481470 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.483952 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.483954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.483954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.483960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.485432 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.485840 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.485841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.485841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.485890 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.486155 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.486155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.486156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.486254 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.486340 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.486643 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.486728 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:12:08.486968 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.478297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41825 (local address 127.0.0.1:45670)
0810 21:12:08.478488 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.478533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.478973 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.479200 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.479203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.479449 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.479650 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.479654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.480289 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.480290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.481167 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.481170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.481490 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.481491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.481599 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.482125 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.482134 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.483858 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.485525 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.485535 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.485538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.485769 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.485961 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.485962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.485962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.486090 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.486333 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.486341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.486483 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.486592 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.486743 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260810 21:12:08.487567 19584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:12:08.489820 19319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } attempt: 2
I20260810 21:12:08.491829 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.482917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.27.117.124:42249)
0810 21:12:08.483138 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.483144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.483171 (+ 27us) server_negotiation.cc:407] Connection header received
0810 21:12:08.483197 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.483199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.483220 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.483314 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.484057 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.484765 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.485838 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.486246 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.486326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.488857 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.488866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.488866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.488873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.490575 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.490927 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.490931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.490931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.491003 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.491219 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.491219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.491220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.491386 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.491583 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.491632 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.491705 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:12:08.495292 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.487823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:45039 (local address 127.0.0.1:60108)
0810 21:12:08.487870 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.487880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.487924 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.488119 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.488120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.488245 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.488291 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.488292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.489028 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.489029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.489709 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.489712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.490278 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.490279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.490356 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.490978 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.490982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.492356 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.494194 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.494195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.494195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.494368 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.494596 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.494597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.494597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.494659 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.494959 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.494960 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.495012 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.495108 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.495199 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 21:12:08.495374 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.487910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.27.117.125:45039)
0810 21:12:08.487983 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.487986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.487994 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:12:08.488018 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.488019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.488028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.488102 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.488368 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.488944 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.489838 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.490148 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.490221 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.492426 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.492427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.492427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.492435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.494087 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.494458 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.494459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.494460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.494521 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.494721 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.494722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.494723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.494862 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.494929 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.495157 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.495249 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:12:08.496132 19584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:08.499274 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.490096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:48998)
0810 21:12:08.490140 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.490151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.490190 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.490412 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.490413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.490610 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.490654 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.490657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.491276 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.491277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.492010 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.492013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.493009 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.493009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.493072 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.493450 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.493454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.494861 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.497574 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.497575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.497576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.497802 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.498016 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.498016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.498017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.498079 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.498865 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.498866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.498916 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.499013 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.499143 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 21:12:08.499717 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.490134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.27.117.124:42249)
0810 21:12:08.490278 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.490279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.490286 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:12:08.490309 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.490310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.490323 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.490399 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.490745 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.491188 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.492289 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.492629 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.493007 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.494942 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.494943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.494943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.494950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.497498 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.497879 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.497880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.497880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.497939 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.498404 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.498406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.498406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.498532 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.498868 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.499473 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.499568 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:12:08.500728 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.492658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.27.117.126:41825)
0810 21:12:08.492694 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.492695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.492760 (+ 65us) server_negotiation.cc:407] Connection header received
0810 21:12:08.492863 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.492864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.492873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.492921 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.493508 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.494167 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.495131 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.495430 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.495480 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.497907 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.497908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.497909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.497915 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.499364 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.499753 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.499755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.499755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.499811 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.500010 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.500010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.500011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.500140 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.500353 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.500557 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.500629 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:12:08.502272 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: Bootstrap starting.
I20260810 21:12:08.503132 19319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:08.503433 19319 log.cc:824] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:08.503916 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.496369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:49008)
0810 21:12:08.496427 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.496440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.496490 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.496673 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.496673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.496881 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.496915 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.496918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.497797 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.497798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.498527 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.498530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.499042 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.499043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.499127 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.499570 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.499574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.500939 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.502854 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.502856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.502856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.503059 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.503265 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.503266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.503267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.503320 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.503606 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.503607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.503644 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.503709 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.503796 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 21:12:08.504196 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.496435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.27.117.124:42249)
0810 21:12:08.496558 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.496560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.496567 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:12:08.496584 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.496584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.496594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.496639 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.497006 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.497714 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.498616 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.498921 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.499009 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.501015 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.501017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.501018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.501026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.502743 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.503134 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.503135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.503136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.503189 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.503394 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.503395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.503395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.503529 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.503602 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.504048 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.504122 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 21:12:08.505725 19584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833: Bootstrap starting.
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:12:08.506390 19584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:08.506678 19584 log.cc:824] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:08.504431 19319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: No bootstrap required, opened a new log
I20260810 21:12:08.508435 19319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.508565 19319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:08.508616 19319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4602737d094e4d91873eaac3df326e99, State: Initialized, Role: FOLLOWER
I20260810 21:12:08.508739 19319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.509093 19319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:08.509049 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.509135 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:12:08.509265 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.501518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.27.117.125:45039)
0810 21:12:08.501554 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.501556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.501595 (+ 39us) server_negotiation.cc:407] Connection header received
0810 21:12:08.501663 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.501664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.501673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.501725 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.502315 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.502833 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.503765 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.504015 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.504065 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.506526 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.506528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.506528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.506536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.508000 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.508366 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.508367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.508367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.508423 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.508657 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.508658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.508658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.508868 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.509053 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.509091 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.509165 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:12:08.510208 19584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833: No bootstrap required, opened a new log
I20260810 21:12:08.510964 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:12:08.511808 19584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.512006 19584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:08.512074 19584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d481877b934056afa63afb1ac08833, State: Initialized, Role: FOLLOWER
I20260810 21:12:08.512204 19584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.512480 19603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.512584 19603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:12:08.512603 19584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 21:12:08.514226 19628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:12:08.514334 19628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 21:12:08.515626 19646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:12:08.515753 19646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:12:08.514853 19202 master_runner.cc:186] Error getting master registration for 127.27.117.125:45039: OK, instance_id { permanent_uuid: "74b65a4a58674e76ab65752723693750" instance_seqno: 1786396328311542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 21:12:08.517403 19202 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:12:08.520625 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.512344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.27.117.125:45039)
0810 21:12:08.512401 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.512402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.512538 (+ 136us) server_negotiation.cc:407] Connection header received
0810 21:12:08.512636 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.512638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.512648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.512677 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.513314 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.514014 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.515145 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.515414 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.515460 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.516999 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.517001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.517001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.517008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.518761 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.519786 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.519787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.519787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.519841 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.520065 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.520066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.520066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.520183 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.520255 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.520455 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.520535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 21:12:08.519727 19456 master_runner.cc:186] Error getting master registration for 127.27.117.125:45039: OK, instance_id { permanent_uuid: "74b65a4a58674e76ab65752723693750" instance_seqno: 1786396328311542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 21:12:08.523844 19456 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:12:08.524811 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.512651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.27.117.126:41825)
0810 21:12:08.512686 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.512688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.513392 (+ 704us) server_negotiation.cc:407] Connection header received
0810 21:12:08.513449 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.513450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.513461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.513502 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.518091 (+ 4589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.518753 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.520009 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.520294 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.520336 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.521957 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.521958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.521958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.521964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.523523 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.523918 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.523919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.523920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.523979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.524221 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.524222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.524222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.524347 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.524427 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.524626 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.524703 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:12:08.530368 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.512518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.27.117.124:42249)
0810 21:12:08.512563 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.512564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.516058 (+ 3494us) server_negotiation.cc:407] Connection header received
0810 21:12:08.516157 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.516158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.516168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.516196 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.520811 (+ 4615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.524114 (+ 3303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.525467 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.525754 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.525830 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.527351 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.527353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.527353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.527361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.529105 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.529510 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.529512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.529512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.529574 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.529805 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.529807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.529807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.529970 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.530113 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.530205 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.530283 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:12:08.547413 19445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } attempt: 3
Aug 10 21:12:08 dist-test-slave-2kcd krb5kdc[19173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:12:08.555943 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.547731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:42249 (local address 127.0.0.1:49040)
0810 21:12:08.547782 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:08.547793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:08.547830 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:08.548045 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:08.548047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:08.548247 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:08.548308 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.548310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.549302 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.549304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.550126 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:08.550130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.550619 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.550620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:08.550684 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.551198 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:08.551203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:08.552815 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:08.554791 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.554792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.554793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.555041 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.555283 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:08.555284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:08.555284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:08.555341 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:08.555637 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:08.555638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:08.555685 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 21:12:08.555732 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 21:12:08.555831 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 21:12:08.556351 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:08.547845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.27.117.124:42249)
0810 21:12:08.547895 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:08.547897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:08.547906 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:08.547929 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:08.547930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:08.547940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:08.548384 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:08.548430 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.549217 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.550219 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:08.550532 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:08.550607 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:08.552948 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:08.552950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:08.552951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:08.552958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:08.554698 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.555133 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.555134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.555134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.555196 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:08.555406 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:08.555407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:08.555407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:08.555540 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:08.556136 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:08.556179 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 21:12:08.556256 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:12:08.558003 19445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: Bootstrap starting.
I20260810 21:12:08.558765 19445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:08.559093 19445 log.cc:824] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:08.560099 19445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: No bootstrap required, opened a new log
I20260810 21:12:08.562227 19445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.562372 19445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:08.562407 19445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b65a4a58674e76ab65752723693750, State: Initialized, Role: FOLLOWER
I20260810 21:12:08.562585 19445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.563791 19445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:08.563753 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.564864 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 21:12:08.565806 19671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:12:08.565861 19671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:12:08.567654 19336 master_runner.cc:263] No leader master found from master 74b65a4a58674e76ab65752723693750
I20260810 21:12:08.567699 19336 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:12:08.625150 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:08.625319 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.625597 19603 leader_election.cc:288] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4602737d094e4d91873eaac3df326e99 (127.27.117.126:41825), 74b65a4a58674e76ab65752723693750 (127.27.117.125:45039)
I20260810 21:12:08.626094 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19d481877b934056afa63afb1ac08833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4602737d094e4d91873eaac3df326e99" is_pre_election: true
I20260810 21:12:08.626255 19265 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d481877b934056afa63afb1ac08833 in term 0.
I20260810 21:12:08.626581 19408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19d481877b934056afa63afb1ac08833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b65a4a58674e76ab65752723693750" is_pre_election: true
I20260810 21:12:08.626722 19408 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19d481877b934056afa63afb1ac08833 in term 0.
I20260810 21:12:08.626961 19485 leader_election.cc:302] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d481877b934056afa63afb1ac08833, 4602737d094e4d91873eaac3df326e99; no voters:
I20260810 21:12:08.627372 19603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:12:08.627430 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:12:08.627454 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:08.628439 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.628579 19603 leader_election.cc:288] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [CANDIDATE]: Term 1 election: Requested vote from peers 4602737d094e4d91873eaac3df326e99 (127.27.117.126:41825), 74b65a4a58674e76ab65752723693750 (127.27.117.125:45039)
I20260810 21:12:08.629052 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19d481877b934056afa63afb1ac08833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4602737d094e4d91873eaac3df326e99"
I20260810 21:12:08.629137 19265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:08.630120 19265 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d481877b934056afa63afb1ac08833 in term 1.
I20260810 21:12:08.630348 19485 leader_election.cc:302] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d481877b934056afa63afb1ac08833, 4602737d094e4d91873eaac3df326e99; no voters:
I20260810 21:12:08.630504 19408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19d481877b934056afa63afb1ac08833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b65a4a58674e76ab65752723693750"
I20260810 21:12:08.630587 19408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:08.631515 19408 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19d481877b934056afa63afb1ac08833 in term 1.
I20260810 21:12:08.631772 19603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:08.631945 19603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [term 1 LEADER]: Becoming Leader. State: Replica: 19d481877b934056afa63afb1ac08833, State: Running, Role: LEADER
I20260810 21:12:08.632037 19603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } }
I20260810 21:12:08.632434 19603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d481877b934056afa63afb1ac08833. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.632519 19603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:08.632869 19676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:08.633114 19676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:08.635349 19408 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [term 1 FOLLOWER]: Refusing update from remote peer 19d481877b934056afa63afb1ac08833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:08.635661 19265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [term 1 FOLLOWER]: Refusing update from remote peer 19d481877b934056afa63afb1ac08833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:08.636220 19603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:08.636376 19603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:08.638782 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d481877b934056afa63afb1ac08833. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.638867 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:12:08.639997 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d481877b934056afa63afb1ac08833. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.640086 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:12:08.642719 19603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.642799 19603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:08.643275 19596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.643370 19596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:12:08.643656 19681 mvcc.cc:205] Tried to move back new op lower bound from 7317079362087436288 to 7317079362077392896. Current Snapshot: MvccSnapshot[applied={T|T < 7317079362087436288}]
I20260810 21:12:08.643922 19683 mvcc.cc:205] Tried to move back new op lower bound from 7317079362087436288 to 7317079362077392896. Current Snapshot: MvccSnapshot[applied={T|T < 7317079362087436288}]
I20260810 21:12:08.644618 19603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.644696 19603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:08.645232 19651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19d481877b934056afa63afb1ac08833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4602737d094e4d91873eaac3df326e99" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41825 } } peers { permanent_uuid: "74b65a4a58674e76ab65752723693750" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 45039 } } peers { permanent_uuid: "19d481877b934056afa63afb1ac08833" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 42249 } } }
I20260810 21:12:08.645313 19651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:12:08.645605 19686 mvcc.cc:205] Tried to move back new op lower bound from 7317079362087436288 to 7317079362077392896. Current Snapshot: MvccSnapshot[applied={T|T < 7317079362087436288}]
I20260810 21:12:08.646653 19676 catalog_manager.cc:1347] Generated new cluster ID: d6530761cbeb44ed9b49cacfcef493ed
I20260810 21:12:08.646700 19676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:12:08.669740 19676 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:12:08.670606 19676 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:08.684937 19676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d481877b934056afa63afb1ac08833: Generated new TSK 0
I20260810 21:12:08.685318 19676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:12:09.514853 19628 catalog_manager.cc:1259] Loaded cluster ID: d6530761cbeb44ed9b49cacfcef493ed
I20260810 21:12:09.514963 19628 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: loading cluster ID for follower catalog manager: success
I20260810 21:12:09.515905 19628 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: acquiring CA information for follower catalog manager: success
I20260810 21:12:09.516254 19628 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4602737d094e4d91873eaac3df326e99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:12:09.566278 19671 catalog_manager.cc:1259] Loaded cluster ID: d6530761cbeb44ed9b49cacfcef493ed
I20260810 21:12:09.566361 19671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: loading cluster ID for follower catalog manager: success
I20260810 21:12:09.567196 19671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: acquiring CA information for follower catalog manager: success
I20260810 21:12:09.567528 19671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74b65a4a58674e76ab65752723693750: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:12:09.732133 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 19202
I20260810 21:12:09.746551 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 19336
I20260810 21:12:09.758378 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 19456
2026-08-10T21:12:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3918 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:12:09 dist-test-slave-2kcd krb5kdc[19706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:12:09 dist-test-slave-2kcd krb5kdc[19706](info): set up 2 sockets
Aug 10 21:12:09 dist-test-slave-2kcd krb5kdc[19706](info): commencing operation
krb5kdc: starting...
W20260810 21:12:11.044519 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:12:11 dist-test-slave-2kcd krb5kdc[19706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:12:11.167616 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:37921
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:37921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:11.249102 19727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:11.249310 19727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:11.249331 19727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:11.250543 19727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:11.250588 19727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:11.250602 19727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:11.250615 19727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:11.250669 19727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:11.252152 19727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:37921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:11.252408 19727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:11.252673 19727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:11.257529 19739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:11.257681 19733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:11.257826 19727 server_base.cc:1034] running on GCE node
W20260810 21:12:11.257531 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:11.258104 19727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:11.258383 19727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:11.259532 19727 hybrid_clock.cc:648] HybridClock initialized: now 1786396331259520 us; error 31 us; skew 500 ppm
Aug 10 21:12:11 dist-test-slave-2kcd krb5kdc[19706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:11.262213 19727 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:11.262706 19727 webserver.cc:460] Webserver started at http://127.27.117.126:46873/ using document root <none> and password file <none>
I20260810 21:12:11.262940 19727 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:11.262985 19727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:11.263135 19727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:11.264178 19727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "7955307784004e90a0ec4cb41eeb7d56"
format_stamp: "Formatted at 2026-08-10 21:12:11 on dist-test-slave-2kcd"
server_key: "96f5709b1d5cb7a0145924c978045365"
server_key_iv: "4df87e4a8527f6762615fe54060b6e20"
server_key_version: "encryptionkey@0"
I20260810 21:12:11.264554 19727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "7955307784004e90a0ec4cb41eeb7d56"
format_stamp: "Formatted at 2026-08-10 21:12:11 on dist-test-slave-2kcd"
server_key: "96f5709b1d5cb7a0145924c978045365"
server_key_iv: "4df87e4a8527f6762615fe54060b6e20"
server_key_version: "encryptionkey@0"
I20260810 21:12:11.265939 19727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:12:11.266633 19748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:11.266831 19727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:11.266927 19727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "7955307784004e90a0ec4cb41eeb7d56"
format_stamp: "Formatted at 2026-08-10 21:12:11 on dist-test-slave-2kcd"
server_key: "96f5709b1d5cb7a0145924c978045365"
server_key_iv: "4df87e4a8527f6762615fe54060b6e20"
server_key_version: "encryptionkey@0"
I20260810 21:12:11.267000 19727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:11.301245 19727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:11.303690 19727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:11.303858 19727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:11.308076 19727 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:37921
I20260810 21:12:11.308132 19827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:37921 every 8 connection(s)
I20260810 21:12:11.308498 19727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:11.309419 19829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:11.311714 19829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56: Bootstrap starting.
I20260810 21:12:11.312304 19829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:11.312619 19829 log.cc:824] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:11.313000 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 19727
I20260810 21:12:11.313097 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:12:11.313264 28117 external_mini_cluster.cc:1428] Setting key bcdf5ab137769d8a3e730ee3522e794f
I20260810 21:12:11.313493 19829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56: No bootstrap required, opened a new log
I20260810 21:12:11.315237 19829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955307784004e90a0ec4cb41eeb7d56" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37921 } }
I20260810 21:12:11.315393 19829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:11.315460 19829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7955307784004e90a0ec4cb41eeb7d56, State: Initialized, Role: FOLLOWER
I20260810 21:12:11.315637 19829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955307784004e90a0ec4cb41eeb7d56" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37921 } }
I20260810 21:12:11.315732 19829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:11.315778 19829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:11.315826 19829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:11.316686 19829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955307784004e90a0ec4cb41eeb7d56" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37921 } }
I20260810 21:12:11.316825 19829 leader_election.cc:302] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7955307784004e90a0ec4cb41eeb7d56; no voters:
I20260810 21:12:11.317045 19829 leader_election.cc:288] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:12:11.317065 19836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:11.317221 19836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [term 1 LEADER]: Becoming Leader. State: Replica: 7955307784004e90a0ec4cb41eeb7d56, State: Running, Role: LEADER
I20260810 21:12:11.317332 19829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:11.317317 19836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955307784004e90a0ec4cb41eeb7d56" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37921 } }
I20260810 21:12:11.317862 19837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7955307784004e90a0ec4cb41eeb7d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955307784004e90a0ec4cb41eeb7d56" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37921 } } }
I20260810 21:12:11.317947 19837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:11.318100 19838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7955307784004e90a0ec4cb41eeb7d56. Latest consensus state: current_term: 1 leader_uuid: "7955307784004e90a0ec4cb41eeb7d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7955307784004e90a0ec4cb41eeb7d56" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 37921 } } }
I20260810 21:12:11.318166 19847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:11.318243 19838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:11.319048 19847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:11.320565 19847 catalog_manager.cc:1347] Generated new cluster ID: 3e078fc1986c4dfc97e60ce087de2479
I20260810 21:12:11.320618 19847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:12:11 dist-test-slave-2kcd krb5kdc[19706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:11.326502 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:11.313739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.27.117.126:37921)
0810 21:12:11.313915 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:11.313925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:11.313962 (+ 37us) server_negotiation.cc:407] Connection header received
0810 21:12:11.315225 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:11.315232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:11.315276 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:11.315414 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:11.316171 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:11.317371 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:11.318444 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:11.319087 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:11.319214 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:11.321995 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:11.322000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:11.322000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:11.322011 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:11.323864 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:11.324267 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:11.324270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:11.324270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:11.324908 (+ 638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:11.325289 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:11.325290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:11.325290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:11.325578 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:11.325727 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:11.326157 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 21:12:11.326295 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260810 21:12:11.327606 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:12:11.329949 19847 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:12:11.330628 19847 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:11.338948 19847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7955307784004e90a0ec4cb41eeb7d56: Generated new TSK 0
I20260810 21:12:11.339121 19847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:12:11.381511 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 19727
2026-08-10T21:12:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1628 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:12:11 dist-test-slave-2kcd krb5kdc[19862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:12:11 dist-test-slave-2kcd krb5kdc[19862](info): set up 2 sockets
Aug 10 21:12:11 dist-test-slave-2kcd krb5kdc[19862](info): commencing operation
krb5kdc: starting...
W20260810 21:12:12.698293 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:12:12 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:12:12.817884 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40667
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:12.899351 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:12.899555 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:12.899577 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:12.900691 19888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:12.900732 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:12.900758 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:12.900830 19888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:12.900861 19888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:12.902289 19888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:12.902525 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:12.902747 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:12.907356 19900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:12.907389 19888 server_base.cc:1034] running on GCE node
W20260810 21:12:12.907563 19897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:12.907389 19898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:12.907828 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:12.908124 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:12.909281 19888 hybrid_clock.cc:648] HybridClock initialized: now 1786396332909266 us; error 39 us; skew 500 ppm
Aug 10 21:12:12 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396332, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:12.911628 19888 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:12.912048 19888 webserver.cc:460] Webserver started at http://127.27.117.126:39043/ using document root <none> and password file <none>
I20260810 21:12:12.912223 19888 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:12.912263 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:12.912350 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:12.913349 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "362d4cf2d0ee4b47b533584e877c8316"
format_stamp: "Formatted at 2026-08-10 21:12:12 on dist-test-slave-2kcd"
server_key: "851110480fabb653566bf4f965362414"
server_key_iv: "f01732ac6c1081cd9d4519dd2922459b"
server_key_version: "encryptionkey@0"
I20260810 21:12:12.913641 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "362d4cf2d0ee4b47b533584e877c8316"
format_stamp: "Formatted at 2026-08-10 21:12:12 on dist-test-slave-2kcd"
server_key: "851110480fabb653566bf4f965362414"
server_key_iv: "f01732ac6c1081cd9d4519dd2922459b"
server_key_version: "encryptionkey@0"
I20260810 21:12:12.914861 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:12:12.915534 19918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:12.915700 19888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:12.915803 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "362d4cf2d0ee4b47b533584e877c8316"
format_stamp: "Formatted at 2026-08-10 21:12:12 on dist-test-slave-2kcd"
server_key: "851110480fabb653566bf4f965362414"
server_key_iv: "f01732ac6c1081cd9d4519dd2922459b"
server_key_version: "encryptionkey@0"
I20260810 21:12:12.915872 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:12.937770 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:12.939577 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:12.939730 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:12.943799 19888 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:40667
I20260810 21:12:12.943841 20008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:40667 every 8 connection(s)
I20260810 21:12:12.944206 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:12.945194 20010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:12.947301 20010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap starting.
I20260810 21:12:12.947822 20010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:12.948060 20010 log.cc:824] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:12.948707 20010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: No bootstrap required, opened a new log
I20260810 21:12:12.950289 20010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:12.950421 20010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:12.950444 20010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362d4cf2d0ee4b47b533584e877c8316, State: Initialized, Role: FOLLOWER
I20260810 21:12:12.950549 20010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:12.950598 20010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:12.950619 20010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:12.950645 20010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:12.951367 20010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:12.951453 20010 leader_election.cc:302] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362d4cf2d0ee4b47b533584e877c8316; no voters:
I20260810 21:12:12.951576 20010 leader_election.cc:288] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:12:12.951634 20015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:12.951778 20015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 LEADER]: Becoming Leader. State: Replica: 362d4cf2d0ee4b47b533584e877c8316, State: Running, Role: LEADER
I20260810 21:12:12.951815 20010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:12.951916 20015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:12.952267 20016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "362d4cf2d0ee4b47b533584e877c8316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } } }
I20260810 21:12:12.952364 20016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:12.952281 20017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362d4cf2d0ee4b47b533584e877c8316. Latest consensus state: current_term: 1 leader_uuid: "362d4cf2d0ee4b47b533584e877c8316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } } }
I20260810 21:12:12.952471 20017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:12.952729 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 19888
I20260810 21:12:12.952860 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:12:12.953030 20030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:12.953068 28117 external_mini_cluster.cc:1428] Setting key af3b3a6225819c797c41ded34f1c0e3e
I20260810 21:12:12.953615 20030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:12.955312 20030 catalog_manager.cc:1347] Generated new cluster ID: 123b68b9dcf644a9971b605bae117cb0
I20260810 21:12:12.955365 20030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:12:12 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:12.964964 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:12.953540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.27.117.126:40667)
0810 21:12:12.953779 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:12.953789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:12.953879 (+ 90us) server_negotiation.cc:407] Connection header received
0810 21:12:12.954243 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:12.954251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:12.954311 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:12.954452 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:12.955290 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:12.956641 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:12.957714 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:12.958353 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:12.958468 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:12.961274 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:12.961279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:12.961279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:12.961291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:12.963297 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:12.963693 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:12.963696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:12.963697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:12.963816 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:12.964021 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:12.964022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:12.964023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:12.964344 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:12.964500 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:12.964598 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 21:12:12.964732 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260810 21:12:12.968505 20030 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:12:12.969211 20030 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:12.984975 20030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Generated new TSK 0
I20260810 21:12:12.985179 20030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:12:13.013244 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40667
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:13.094964 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:13.095170 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:13.095192 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:13.096320 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:13.096360 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:13.096386 20054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:13.097952 20054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:13.098169 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:13.098381 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:13.103396 20062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.103435 20054 server_base.cc:1034] running on GCE node
W20260810 21:12:13.103375 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.103389 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.103721 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:13.103979 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:13.105122 20054 hybrid_clock.cc:648] HybridClock initialized: now 1786396333105103 us; error 36 us; skew 500 ppm
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:13.107502 20054 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:13.107918 20054 webserver.cc:460] Webserver started at http://127.27.117.65:36621/ using document root <none> and password file <none>
I20260810 21:12:13.108152 20054 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:13.108229 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:13.108361 20054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:13.109434 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "092430078ca5431e89bca6157a32ba5a"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "b3b0abeb8434190d5bf0fed2620836ab"
server_key_iv: "5f96013bd85f647c00c5b5b6b465e825"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.109792 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "092430078ca5431e89bca6157a32ba5a"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "b3b0abeb8434190d5bf0fed2620836ab"
server_key_iv: "5f96013bd85f647c00c5b5b6b465e825"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.111056 20054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:12:13.111765 20082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.111930 20054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:13.112041 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "092430078ca5431e89bca6157a32ba5a"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "b3b0abeb8434190d5bf0fed2620836ab"
server_key_iv: "5f96013bd85f647c00c5b5b6b465e825"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.112114 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:13.125025 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:13.126718 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:13.126869 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:13.127085 20054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:13.127393 20054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:13.127442 20054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.127472 20054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:13.127497 20054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.133375 20054 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37859
I20260810 21:12:13.133447 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37859 every 8 connection(s)
I20260810 21:12:13.133765 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:12:13.138007 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 20054
I20260810 21:12:13.138090 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:12:13.138280 28117 external_mini_cluster.cc:1428] Setting key 999a81c1ae1e332771dad4f848221c81
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
I20260810 21:12:13.148001 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.134448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:58779 (local address 127.27.117.126:40667)
0810 21:12:13.134495 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.134497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.134716 (+ 219us) server_negotiation.cc:407] Connection header received
0810 21:12:13.135243 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.135244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.135257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.135308 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:13.136240 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.136758 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.137809 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.138026 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.138104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.145055 (+ 6951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:13.145056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:13.145057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:13.145065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:13.146534 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.147034 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.147035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.147035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.147089 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.147360 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.147360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.147360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.147466 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:13.147527 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.147770 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.147863 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:12:13.148123 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.134428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:58779)
0810 21:12:13.134595 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.134647 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.135135 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.135378 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.135382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.135950 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:13.136143 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.136148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.136903 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.136904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.137704 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.137707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.138175 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.138177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.138286 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.143147 (+ 4861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:13.143156 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:13.144953 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:13.146624 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:13.146632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:13.146634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:13.146950 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:13.147159 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:13.147160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:13.147160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:13.147300 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:13.147525 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:13.147531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:13.147659 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.147744 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.147882 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260810 21:12:13.148655 20285 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40667
I20260810 21:12:13.148769 20285 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:13.148983 20285 heartbeater.cc:511] Master 127.27.117.126:40667 requested a full tablet report, sending...
I20260810 21:12:13.149626 19947 ts_manager.cc:194] Registered new tserver with Master: 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859)
I20260810 21:12:13.150461 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:58779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:12:13.166878 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40667
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:13.249212 20296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:13.249403 20296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:13.249426 20296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:13.250579 20296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:13.250622 20296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:13.250648 20296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:13.252156 20296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:13.252378 20296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:13.252611 20296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:13.257513 20303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.257522 20307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.257676 20296 server_base.cc:1034] running on GCE node
W20260810 21:12:13.257526 20305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.257886 20296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:13.258113 20296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:13.259282 20296 hybrid_clock.cc:648] HybridClock initialized: now 1786396333259269 us; error 32 us; skew 500 ppm
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:13.261694 20296 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:13.262140 20296 webserver.cc:460] Webserver started at http://127.27.117.66:46675/ using document root <none> and password file <none>
I20260810 21:12:13.262347 20296 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:13.262391 20296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:13.262537 20296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:13.263499 20296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "98fda818aae54e15a0d351ae5eb38a75"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "1d1b1ae8e1d7a0422be70458efa60fe2"
server_key_iv: "0e6f6eddbe3fe7ecd30448f4b2bb5582"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.263844 20296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "98fda818aae54e15a0d351ae5eb38a75"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "1d1b1ae8e1d7a0422be70458efa60fe2"
server_key_iv: "0e6f6eddbe3fe7ecd30448f4b2bb5582"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.265309 20296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 21:12:13.265991 20317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.266156 20296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:13.266240 20296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "98fda818aae54e15a0d351ae5eb38a75"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "1d1b1ae8e1d7a0422be70458efa60fe2"
server_key_iv: "0e6f6eddbe3fe7ecd30448f4b2bb5582"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.266294 20296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:13.289853 20296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:13.291548 20296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:13.291680 20296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:13.291851 20296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:13.292176 20296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:13.292214 20296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.292234 20296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:13.292279 20296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.297796 20296 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:41851
I20260810 21:12:13.297874 20516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:41851 every 8 connection(s)
I20260810 21:12:13.298156 20296 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:12:13.302352 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 20296
I20260810 21:12:13.302459 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:12:13.302659 28117 external_mini_cluster.cc:1428] Setting key 373130c2cbfd8a6801cd2e72c58c25c8
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:13.314465 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.298846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:55395 (local address 127.27.117.126:40667)
0810 21:12:13.298889 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.298891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.299081 (+ 190us) server_negotiation.cc:407] Connection header received
0810 21:12:13.299518 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.299519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.299530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.299580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:13.300430 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.300991 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.301889 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.302073 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.302123 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.304745 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:13.304747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:13.304747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:13.304754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:13.313010 (+ 8256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.313486 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.313487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.313488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.313559 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.313808 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.313809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.313810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.313925 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:13.313996 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.314285 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.314375 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:12:13.314677 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.298799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:55395)
0810 21:12:13.298954 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.298992 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.299417 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.299635 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.299638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.300171 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:13.300344 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.300348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.301072 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.301073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.301818 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.301819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.302143 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.302144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.302240 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.302719 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:13.302726 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:13.304624 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:13.313120 (+ 8496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:13.313128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:13.313130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:13.313406 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:13.313631 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:13.313632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:13.313633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:13.313736 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:13.314006 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:13.314014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:13.314139 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.314259 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.314411 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260810 21:12:13.315302 20517 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40667
I20260810 21:12:13.315380 20517 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:13.315519 20517 heartbeater.cc:511] Master 127.27.117.126:40667 requested a full tablet report, sending...
I20260810 21:12:13.315979 19947 ts_manager.cc:194] Registered new tserver with Master: 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851)
I20260810 21:12:13.316522 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:55395
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:12:13.334709 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40667
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:13.415592 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:13.415791 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:13.415814 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:13.417045 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:13.417098 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:13.417136 20529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:13.418911 20529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:13.419190 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:13.419425 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:13.424328 20545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.424388 20542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.424328 20541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.424700 20529 server_base.cc:1034] running on GCE node
I20260810 21:12:13.424944 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:13.425192 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:13.426333 20529 hybrid_clock.cc:648] HybridClock initialized: now 1786396333426314 us; error 34 us; skew 500 ppm
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:13.428731 20529 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:13.429224 20529 webserver.cc:460] Webserver started at http://127.27.117.67:35547/ using document root <none> and password file <none>
I20260810 21:12:13.429447 20529 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:13.429512 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:13.429672 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:13.430658 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "b49afa89e4d944f2a74c5953f320dd7d"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "dace31efac2bdf586c01e32b7828e3cc"
server_key_iv: "9cb5d52e7cdb4871084b8253b4882a7c"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.431021 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "b49afa89e4d944f2a74c5953f320dd7d"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "dace31efac2bdf586c01e32b7828e3cc"
server_key_iv: "9cb5d52e7cdb4871084b8253b4882a7c"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.432307 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:12:13.433040 20563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.433209 20529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:12:13.433307 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "b49afa89e4d944f2a74c5953f320dd7d"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "dace31efac2bdf586c01e32b7828e3cc"
server_key_iv: "9cb5d52e7cdb4871084b8253b4882a7c"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.433382 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:13.456339 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:13.458326 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:13.458488 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:13.458694 20529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:13.459018 20529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:13.459079 20529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.459120 20529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:13.459147 20529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.464922 20529 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:46407
I20260810 21:12:13.464982 20759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:46407 every 8 connection(s)
I20260810 21:12:13.465303 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:12:13.469769 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 20529
I20260810 21:12:13.469879 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:12:13.470062 28117 external_mini_cluster.cc:1428] Setting key f0e41bc58601f572462bc9015202c9e6
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:13.474115 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.465939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:42239 (local address 127.27.117.126:40667)
0810 21:12:13.465979 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.465981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.466148 (+ 167us) server_negotiation.cc:407] Connection header received
0810 21:12:13.466644 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.466645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.466659 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.466709 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:13.467597 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.468099 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.469196 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.469390 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.469453 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.471734 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:13.471735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:13.471735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:13.471741 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:13.473016 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.473378 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.473379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.473379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.473428 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.473628 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.473629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.473629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.473724 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:13.473775 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.473994 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.474054 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:12:13.474381 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.465903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:42239)
0810 21:12:13.466035 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.466076 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.466536 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.466768 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.466772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.467312 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:13.467505 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.467509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.468194 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.468195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.469102 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.469104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.469468 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.469469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.469573 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.470155 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:13.470165 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:13.471650 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:13.473086 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:13.473093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:13.473094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:13.473311 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:13.473482 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:13.473482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:13.473482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:13.473577 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:13.473778 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:13.473782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:13.473886 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.473966 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.474088 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260810 21:12:13.474884 20760 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40667
I20260810 21:12:13.474997 20760 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:13.475179 20760 heartbeater.cc:511] Master 127.27.117.126:40667 requested a full tablet report, sending...
I20260810 21:12:13.475534 19947 ts_manager.cc:194] Registered new tserver with Master: b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
I20260810 21:12:13.475909 19947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:42239
I20260810 21:12:13.481580 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:12:13.488761 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.482875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.27.117.126:40667)
0810 21:12:13.482950 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.482951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.483042 (+ 91us) server_negotiation.cc:407] Connection header received
0810 21:12:13.483132 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.483133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.483143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.483189 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:13.483743 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.484219 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.484978 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.485197 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.485270 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.486423 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:13.486424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:13.486425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:13.486432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:13.487608 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.487939 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.487940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.487941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.488007 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:13.488199 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:13.488199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:13.488200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:13.488346 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:13.488448 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.488586 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.488663 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:12:13.490171 19947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:12:13.491729 19947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:12:13.498042 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.493969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:41851 (local address 127.0.0.1:60524)
0810 21:12:13.494143 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.494184 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.494252 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.494671 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.494674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.494676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:13.494741 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.494743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.495762 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.495763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.496939 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.496942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.497752 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.497753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.497832 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.497844 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.497912 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.497964 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260810 21:12:13.498054 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.494192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.27.117.66:41851)
0810 21:12:13.494510 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.494514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.494522 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:12:13.494564 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.494566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.494577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.494666 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:13.494829 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.495659 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.497042 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.497668 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.497751 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.497811 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.497933 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.497973 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":91,"threads_started":1}
I20260810 21:12:13.498276 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.493976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46407 (local address 127.0.0.1:33132)
0810 21:12:13.494381 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.494389 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.494424 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.494720 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.494721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.494723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:13.494782 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.494784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.496152 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.496153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.497183 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.497185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.498051 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.498053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.498126 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.498131 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.498163 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.498203 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":114,"threads_started":1}
I20260810 21:12:13.498447 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.493969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37859 (local address 127.0.0.1:51756)
0810 21:12:13.494240 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.494253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.494309 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.495004 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.495005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.495007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:13.495048 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.495050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.496281 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.496282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.497316 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.497319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.498178 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.498179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.498247 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.498251 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.498335 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.498374 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"thread_start_us":112,"threads_started":1}
I20260810 21:12:13.498502 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.494190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.27.117.65:37859)
0810 21:12:13.494833 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.494837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.494847 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:12:13.494887 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.494889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.494916 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.494995 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:13.495121 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.496174 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.497402 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.498112 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.498178 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.498228 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.498350 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.498398 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":91,"threads_started":1}
I20260810 21:12:13.498955 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.494379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.27.117.67:46407)
0810 21:12:13.494544 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.494549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.494559 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:12:13.494596 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.494598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.494614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.494866 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:13.494951 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.496046 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.497256 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.497944 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.498746 (+ 802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.498792 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.498812 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.498871 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260810 21:12:13.498867 20400 tablet_service.cc:1467] Processing CreateTablet for tablet c0513e892bfa49608059fe3e7c6c544e (DEFAULT_TABLE table=test-table [id=7e05db8855894d2f8f504a5e2c214d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:13.499251 20400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0513e892bfa49608059fe3e7c6c544e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:13.499415 20157 tablet_service.cc:1467] Processing CreateTablet for tablet c0513e892bfa49608059fe3e7c6c544e (DEFAULT_TABLE table=test-table [id=7e05db8855894d2f8f504a5e2c214d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:13.499490 20641 tablet_service.cc:1467] Processing CreateTablet for tablet c0513e892bfa49608059fe3e7c6c544e (DEFAULT_TABLE table=test-table [id=7e05db8855894d2f8f504a5e2c214d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:13.499744 20641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0513e892bfa49608059fe3e7c6c544e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:13.499729 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0513e892bfa49608059fe3e7c6c544e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:13.502264 20799 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Bootstrap starting.
I20260810 21:12:13.502737 20800 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Bootstrap starting.
I20260810 21:12:13.502911 20799 tablet_bootstrap.cc:654] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:13.503245 20801 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Bootstrap starting.
I20260810 21:12:13.503409 20800 tablet_bootstrap.cc:654] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:13.503499 20799 log.cc:824] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:13.503901 20801 tablet_bootstrap.cc:654] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:13.503933 20800 log.cc:824] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:13.504355 20799 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: No bootstrap required, opened a new log
I20260810 21:12:13.504442 20799 ts_tablet_manager.cc:1397] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:13.504462 20801 log.cc:824] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:13.504698 20800 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: No bootstrap required, opened a new log
I20260810 21:12:13.504774 20800 ts_tablet_manager.cc:1397] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:13.505300 20801 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: No bootstrap required, opened a new log
I20260810 21:12:13.505425 20801 ts_tablet_manager.cc:1397] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:13.506150 20800 raft_consensus.cc:348] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.506314 20800 raft_consensus.cc:374] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:13.506361 20800 raft_consensus.cc:729] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98fda818aae54e15a0d351ae5eb38a75, State: Initialized, Role: FOLLOWER
I20260810 21:12:13.506489 20800 consensus_queue.cc:260] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.506560 20799 raft_consensus.cc:348] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.506726 20799 raft_consensus.cc:374] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:13.506774 20799 raft_consensus.cc:729] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49afa89e4d944f2a74c5953f320dd7d, State: Initialized, Role: FOLLOWER
I20260810 21:12:13.506784 20800 ts_tablet_manager.cc:1428] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:13.506901 20517 heartbeater.cc:503] Master 127.27.117.126:40667 was elected leader, sending a full tablet report...
I20260810 21:12:13.506882 20799 consensus_queue.cc:260] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.507174 20760 heartbeater.cc:503] Master 127.27.117.126:40667 was elected leader, sending a full tablet report...
I20260810 21:12:13.507157 20799 ts_tablet_manager.cc:1428] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:12:13.507740 20801 raft_consensus.cc:348] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.507896 20801 raft_consensus.cc:374] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:13.507943 20801 raft_consensus.cc:729] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092430078ca5431e89bca6157a32ba5a, State: Initialized, Role: FOLLOWER
I20260810 21:12:13.508064 20801 consensus_queue.cc:260] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.508351 20801 ts_tablet_manager.cc:1428] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 21:12:13.508374 20285 heartbeater.cc:503] Master 127.27.117.126:40667 was elected leader, sending a full tablet report...
W20260810 21:12:13.548743 20521 tablet.cc:2369] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:12:13.603056 20807 raft_consensus.cc:482] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:13.603193 20807 raft_consensus.cc:504] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.603538 20807 leader_election.cc:288] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
I20260810 21:12:13.606352 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.603737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46407 (local address 127.27.117.66:51697)
0810 21:12:13.603784 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.603796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.603841 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.604038 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.604040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.604042 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:13.604118 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.604119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.604754 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.604755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.605503 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.605505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.606090 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.606091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.606174 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.606177 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.606241 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.606282 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 21:12:13.606352 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.603802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:51697 (local address 127.27.117.67:46407)
0810 21:12:13.603867 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.603869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.603881 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:12:13.603914 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.603915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.603923 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.604022 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:13.604194 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.604672 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.605605 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.606005 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.606090 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.606099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.606246 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.606282 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:12:13.606628 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.603955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:59301 (local address 127.27.117.65:37859)
0810 21:12:13.604004 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.604006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.604113 (+ 107us) server_negotiation.cc:407] Connection header received
0810 21:12:13.604200 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.604201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.604210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.604266 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:13.604407 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.604953 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.605860 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.606240 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.606323 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.606331 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.606514 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.606562 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 21:12:13.606746 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0513e892bfa49608059fe3e7c6c544e" candidate_uuid: "98fda818aae54e15a0d351ae5eb38a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49afa89e4d944f2a74c5953f320dd7d" is_pre_election: true
I20260810 21:12:13.606855 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.603739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37859 (local address 127.27.117.66:59301)
0810 21:12:13.604050 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.604058 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.604126 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:13.604271 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:13.604271 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:13.604272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:13.604342 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.604344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.605042 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.605043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.605767 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:13.605769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.606325 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.606326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:13.606397 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.606401 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:13.606728 (+ 327us) client_negotiation.cc:240] Negotiation successful
0810 21:12:13.606778 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":54,"thread_start_us":115,"threads_started":1}
I20260810 21:12:13.606911 20679 raft_consensus.cc:2454] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98fda818aae54e15a0d351ae5eb38a75 in term 0.
I20260810 21:12:13.607159 20321 leader_election.cc:302] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d; no voters:
I20260810 21:12:13.607168 20200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0513e892bfa49608059fe3e7c6c544e" candidate_uuid: "98fda818aae54e15a0d351ae5eb38a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "092430078ca5431e89bca6157a32ba5a" is_pre_election: true
I20260810 21:12:13.607301 20807 raft_consensus.cc:2781] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:12:13.607316 20200 raft_consensus.cc:2454] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98fda818aae54e15a0d351ae5eb38a75 in term 0.
I20260810 21:12:13.607363 20807 raft_consensus.cc:482] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:12:13.607481 20807 raft_consensus.cc:3037] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:13.608248 20807 raft_consensus.cc:504] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.608403 20807 leader_election.cc:288] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [CANDIDATE]: Term 1 election: Requested vote from peers 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
I20260810 21:12:13.608587 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0513e892bfa49608059fe3e7c6c544e" candidate_uuid: "98fda818aae54e15a0d351ae5eb38a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49afa89e4d944f2a74c5953f320dd7d"
I20260810 21:12:13.608609 20200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0513e892bfa49608059fe3e7c6c544e" candidate_uuid: "98fda818aae54e15a0d351ae5eb38a75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "092430078ca5431e89bca6157a32ba5a"
I20260810 21:12:13.608662 20679 raft_consensus.cc:3037] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:13.608695 20200 raft_consensus.cc:3037] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:13.609395 20679 raft_consensus.cc:2454] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98fda818aae54e15a0d351ae5eb38a75 in term 1.
I20260810 21:12:13.609549 20200 raft_consensus.cc:2454] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98fda818aae54e15a0d351ae5eb38a75 in term 1.
I20260810 21:12:13.609565 20321 leader_election.cc:302] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d; no voters:
I20260810 21:12:13.609722 20807 raft_consensus.cc:2781] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:13.609901 20807 raft_consensus.cc:686] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 1 LEADER]: Becoming Leader. State: Replica: 98fda818aae54e15a0d351ae5eb38a75, State: Running, Role: LEADER
I20260810 21:12:13.610002 20807 consensus_queue.cc:237] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:13.610806 19947 catalog_manager.cc:5495] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66). New cstate: current_term: 1 leader_uuid: "98fda818aae54e15a0d351ae5eb38a75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260810 21:12:13.616583 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.614420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.27.117.66:41851)
0810 21:12:13.614459 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.614460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.614469 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:13.614540 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:13.614541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:13.614555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:13.614610 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:13.614714 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.615120 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.615970 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:13.616146 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:13.616196 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:13.616328 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:13.616415 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:13.616467 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:13.616498 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 21:12:13.616513 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:12:13.618221 20200 raft_consensus.cc:1261] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Refusing update from remote peer 98fda818aae54e15a0d351ae5eb38a75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:13.618247 20679 raft_consensus.cc:1261] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Refusing update from remote peer 98fda818aae54e15a0d351ae5eb38a75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:13.618554 20818 consensus_queue.cc:1035] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:13.618690 20807 consensus_queue.cc:1035] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:13.621047 20824 mvcc.cc:205] Tried to move back new op lower bound from 7317079382497832960 to 7317079382467072000. Current Snapshot: MvccSnapshot[applied={T|T < 7317079382497832960}]
I20260810 21:12:13.621824 20827 mvcc.cc:205] Tried to move back new op lower bound from 7317079382497832960 to 7317079382467072000. Current Snapshot: MvccSnapshot[applied={T|T < 7317079382497832960 or (T in {7317079382497832960})}]
I20260810 21:12:13.623387 19947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:13.623483 19947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:13.624181 19947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Sending DeleteTablet for 3 replicas of tablet c0513e892bfa49608059fe3e7c6c544e
I20260810 21:12:13.624512 20400 tablet_service.cc:1514] Processing DeleteTablet for tablet c0513e892bfa49608059fe3e7c6c544e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:13 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60524
I20260810 21:12:13.624526 20641 tablet_service.cc:1514] Processing DeleteTablet for tablet c0513e892bfa49608059fe3e7c6c544e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:13 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33132
I20260810 21:12:13.624745 20838 tablet_replica.cc:330] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: stopping tablet replica
I20260810 21:12:13.624747 20157 tablet_service.cc:1514] Processing DeleteTablet for tablet c0513e892bfa49608059fe3e7c6c544e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:13 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:51756
I20260810 21:12:13.624866 20838 raft_consensus.cc:2229] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:13.624972 20838 raft_consensus.cc:2258] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:13.625051 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 20054
I20260810 21:12:13.625062 20837 tablet_replica.cc:330] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: stopping tablet replica
I20260810 21:12:13.633200 20837 raft_consensus.cc:2229] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:12:13.633292 20837 raft_consensus.cc:2258] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:13.637334 20837 ts_tablet_manager.cc:1905] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:13.639169 20837 ts_tablet_manager.cc:1918] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:13.639298 20837 log.cc:1196] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/c0513e892bfa49608059fe3e7c6c544e
I20260810 21:12:13.639588 20837 ts_tablet_manager.cc:1939] T c0513e892bfa49608059fe3e7c6c544e P 98fda818aae54e15a0d351ae5eb38a75: Deleting consensus metadata
I20260810 21:12:13.640055 19924 catalog_manager.cc:4841] TS 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): tablet c0513e892bfa49608059fe3e7c6c544e (table test-table [id=7e05db8855894d2f8f504a5e2c214d14]) successfully deleted
I20260810 21:12:13.652020 20838 ts_tablet_manager.cc:1905] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:13.654184 20838 ts_tablet_manager.cc:1918] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:13.654577 20838 log.cc:1196] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/c0513e892bfa49608059fe3e7c6c544e
I20260810 21:12:13.655031 20838 ts_tablet_manager.cc:1939] T c0513e892bfa49608059fe3e7c6c544e P b49afa89e4d944f2a74c5953f320dd7d: Deleting consensus metadata
I20260810 21:12:13.655833 19920 catalog_manager.cc:4841] TS b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): tablet c0513e892bfa49608059fe3e7c6c544e (table test-table [id=7e05db8855894d2f8f504a5e2c214d14]) successfully deleted
W20260810 21:12:13.660765 19926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:37859 (error 108)
I20260810 21:12:13.661108 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 20296
I20260810 21:12:13.661970 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.661331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37859 (local address 127.0.0.1:51772)
0810 21:12:13.661880 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.661925 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527}
W20260810 21:12:13.662143 19926 catalog_manager.cc:4584] TS 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0513e892bfa49608059fe3e7c6c544e: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37859: connect: Connection refused (error 111)
I20260810 21:12:13.695729 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 20529
I20260810 21:12:13.705865 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 19888
I20260810 21:12:13.717006 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40667
--webserver_interface=127.27.117.126
--webserver_port=39043
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:13.805238 20842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:13.805539 20842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:13.805579 20842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:13.807132 20842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:13.807194 20842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:13.807216 20842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:13.807236 20842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:13.807265 20842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:13.809237 20842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=39043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:13.809484 20842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:13.809710 20842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:13.822625 20854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.822625 20855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.822804 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.823035 20842 server_base.cc:1034] running on GCE node
I20260810 21:12:13.823372 20842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:13.823719 20842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:13.824843 20842 hybrid_clock.cc:648] HybridClock initialized: now 1786396333824818 us; error 46 us; skew 500 ppm
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:13.827669 20842 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:13.828249 20842 webserver.cc:460] Webserver started at http://127.27.117.126:39043/ using document root <none> and password file <none>
I20260810 21:12:13.828536 20842 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:13.828645 20842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:13.830363 20842 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:13.831131 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.831306 20842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:13.831422 20842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "362d4cf2d0ee4b47b533584e877c8316"
format_stamp: "Formatted at 2026-08-10 21:12:12 on dist-test-slave-2kcd"
server_key: "851110480fabb653566bf4f965362414"
server_key_iv: "f01732ac6c1081cd9d4519dd2922459b"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.831844 20842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:13.841590 20842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:13.843669 20842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:13.843842 20842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:13.847960 20842 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:40667
I20260810 21:12:13.848031 20962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:40667 every 8 connection(s)
I20260810 21:12:13.848408 20842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:13.850257 20964 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:12:13.850950 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap starting.
I20260810 21:12:13.852283 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 20842
I20260810 21:12:13.852588 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:37859
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:40667
--webserver_port=36621
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:13.860930 20964 log.cc:824] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:13.862849 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:13.863185 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap complete.
I20260810 21:12:13.865995 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:13.866236 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362d4cf2d0ee4b47b533584e877c8316, State: Initialized, Role: FOLLOWER
I20260810 21:12:13.866407 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:13.866468 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:13.866521 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:13.866597 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 1 FOLLOWER]: Advancing to term 2
I20260810 21:12:13.867717 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:13.867830 20964 leader_election.cc:302] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362d4cf2d0ee4b47b533584e877c8316; no voters:
I20260810 21:12:13.868052 20964 leader_election.cc:288] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 21:12:13.868088 20971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 21:12:13.868206 20971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 LEADER]: Becoming Leader. State: Replica: 362d4cf2d0ee4b47b533584e877c8316, State: Running, Role: LEADER
I20260810 21:12:13.868305 20971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:13.868335 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:13.868614 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362d4cf2d0ee4b47b533584e877c8316. Latest consensus state: current_term: 2 leader_uuid: "362d4cf2d0ee4b47b533584e877c8316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } } }
I20260810 21:12:13.868688 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:13.868855 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "362d4cf2d0ee4b47b533584e877c8316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } } }
I20260810 21:12:13.868937 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:13.869305 20978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:13.870180 20978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:13.870292 20978 catalog_manager.cc:1259] Loaded cluster ID: 123b68b9dcf644a9971b605bae117cb0
I20260810 21:12:13.870337 20978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:12:13.870674 20978 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 21:12:13.873450 20978 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 21:12:13.875276 20978 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:13.875463 20978 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Loaded TSK: 0
I20260810 21:12:13.875669 20978 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 21:12:13.947956 20968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:13.948156 20968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:13.948180 20968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:13.949364 20968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:13.949406 20968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:13.949432 20968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:13.951224 20968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:37859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=36621
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:13.951470 20968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:13.951709 20968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:13.957479 21013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.957499 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:13.957515 21016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:13.957916 20968 server_base.cc:1034] running on GCE node
I20260810 21:12:13.958122 20968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:13.958410 20968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:13.959602 20968 hybrid_clock.cc:648] HybridClock initialized: now 1786396333959587 us; error 38 us; skew 500 ppm
Aug 10 21:12:13 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:13.962597 20968 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:13.963164 20968 webserver.cc:460] Webserver started at http://127.27.117.65:36621/ using document root <none> and password file <none>
I20260810 21:12:13.963419 20968 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:13.963507 20968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:13.965305 20968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:12:13.966113 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:13.966284 20968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:13.966391 20968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "092430078ca5431e89bca6157a32ba5a"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "b3b0abeb8434190d5bf0fed2620836ab"
server_key_iv: "5f96013bd85f647c00c5b5b6b465e825"
server_key_version: "encryptionkey@0"
I20260810 21:12:13.966722 20968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:13.982250 20968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:13.984514 20968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:13.984661 20968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:13.984936 20968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:13.985394 21038 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 21:12:13.987243 20968 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 21:12:13.987301 20968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 21:12:13.987345 20968 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 21:12:13.987973 20968 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 21:12:13.988039 20968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 21:12:13.988108 21038 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Bootstrap starting.
I20260810 21:12:13.995581 20968 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37859
I20260810 21:12:13.995687 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37859 every 8 connection(s)
I20260810 21:12:13.995980 20968 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:12:13.998804 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 20968
I20260810 21:12:13.999085 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:41851
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:40667
--webserver_port=46675
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:13.999375 21038 log.cc:824] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:14.001978 21038 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:14.002290 21038 tablet_bootstrap.cc:492] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Bootstrap complete.
I20260810 21:12:14.003603 21038 ts_tablet_manager.cc:1397] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.012s
I20260810 21:12:14.004992 21038 raft_consensus.cc:348] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:14.005384 21038 raft_consensus.cc:729] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092430078ca5431e89bca6157a32ba5a, State: Initialized, Role: FOLLOWER
I20260810 21:12:14.005545 21038 consensus_queue.cc:260] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } }
I20260810 21:12:14.006168 21038 ts_tablet_manager.cc:1428] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396333, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:14.013237 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.996648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:33591)
0810 21:12:13.996845 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:13.996887 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:13.997367 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.001817 (+ 4450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.001822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.002216 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:14.002521 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.002528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.003636 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.003638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.004477 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.004480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.005369 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.005371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.005521 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.006171 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:14.006191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:14.008320 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:14.011056 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:14.011064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:14.011066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:14.011393 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:14.011737 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:14.011738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:14.011738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:14.011885 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:14.012416 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:14.012424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:14.012591 (+ 167us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.012700 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.012942 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260810 21:12:14.014233 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:13.996940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:33591 (local address 127.27.117.126:40667)
0810 21:12:13.999809 (+ 2869us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:13.999817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:13.999860 (+ 43us) server_negotiation.cc:407] Connection header received
0810 21:12:14.001518 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.001527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.001593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.002443 (+ 850us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:14.002631 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.003532 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.004580 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.005249 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.006015 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.008481 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:14.008486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:14.008486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:14.008498 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:14.010892 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.011503 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.011506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.011506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.011646 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.011971 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.011972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.011972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.012314 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:14.013724 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.013808 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.013964 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2819,"thread_start_us":85,"threads_started":1}
I20260810 21:12:14.015313 21230 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40667
I20260810 21:12:14.015460 21230 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:14.015659 21230 heartbeater.cc:511] Master 127.27.117.126:40667 requested a full tablet report, sending...
I20260810 21:12:14.016537 20905 ts_manager.cc:194] Registered new tserver with Master: 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859)
I20260810 21:12:14.017880 20905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:33591
I20260810 21:12:14.019129 21230 heartbeater.cc:503] Master 127.27.117.126:40667 was elected leader, sending a full tablet report...
I20260810 21:12:14.028813 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.018299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37859 (local address 127.0.0.1:51784)
0810 21:12:14.018461 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.018496 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.018568 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.025048 (+ 6480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.025051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.025053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:14.025124 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.025127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.026443 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.026444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.027544 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.027547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.028463 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.028465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.028563 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.028577 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.028661 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.028712 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260810 21:12:14.028833 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.018455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.27.117.65:37859)
0810 21:12:14.024873 (+ 6418us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.024882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.024891 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:14.024921 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.024923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.024936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.025032 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:14.025211 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.026350 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.027631 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.028372 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.028448 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.028508 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.028686 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.028728 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6375,"thread_start_us":77,"threads_started":1}
I20260810 21:12:14.029717 21107 tablet_service.cc:1514] Processing DeleteTablet for tablet c0513e892bfa49608059fe3e7c6c544e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:13 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:51784
I20260810 21:12:14.030279 21241 tablet_replica.cc:330] stopping tablet replica
I20260810 21:12:14.030395 21241 raft_consensus.cc:2229] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:14.030465 21241 raft_consensus.cc:2258] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:14.030778 21241 ts_tablet_manager.cc:1905] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:14.033253 21241 ts_tablet_manager.cc:1918] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:14.033334 21241 log.cc:1196] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/c0513e892bfa49608059fe3e7c6c544e
I20260810 21:12:14.033594 21241 ts_tablet_manager.cc:1939] T c0513e892bfa49608059fe3e7c6c544e P 092430078ca5431e89bca6157a32ba5a: Deleting consensus metadata
I20260810 21:12:14.034113 20879 catalog_manager.cc:4841] TS 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859): tablet c0513e892bfa49608059fe3e7c6c544e (table test-table [id=7e05db8855894d2f8f504a5e2c214d14]) successfully deleted
W20260810 21:12:14.096179 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:14.096367 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:14.096390 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:14.097549 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:14.097591 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:14.097617 21237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:14.099031 21237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:41851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=46675
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:14.099261 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:14.099486 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:14.104400 21252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.104414 21251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.104465 21237 server_base.cc:1034] running on GCE node
W20260810 21:12:14.104462 21260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.104823 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:14.105103 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:14.106256 21237 hybrid_clock.cc:648] HybridClock initialized: now 1786396334106238 us; error 40 us; skew 500 ppm
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:14.108665 21237 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:14.109187 21237 webserver.cc:460] Webserver started at http://127.27.117.66:46675/ using document root <none> and password file <none>
I20260810 21:12:14.109452 21237 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:14.109534 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:14.110801 21237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:14.111374 21270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.111542 21237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.111644 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "98fda818aae54e15a0d351ae5eb38a75"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "1d1b1ae8e1d7a0422be70458efa60fe2"
server_key_iv: "0e6f6eddbe3fe7ecd30448f4b2bb5582"
server_key_version: "encryptionkey@0"
I20260810 21:12:14.111963 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:14.120945 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:14.122604 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:14.122810 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:14.123046 21237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:14.123344 21237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:14.123395 21237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.123426 21237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:14.123454 21237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.129300 21237 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:41851
I20260810 21:12:14.129369 21476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:41851 every 8 connection(s)
I20260810 21:12:14.129717 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:12:14.129962 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 21237
I20260810 21:12:14.130230 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:46407
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:40667
--webserver_port=35547
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:14.139632 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.130647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:50567 (local address 127.27.117.126:40667)
0810 21:12:14.130697 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.130699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.130794 (+ 95us) server_negotiation.cc:407] Connection header received
0810 21:12:14.131263 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.131264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.131280 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.131352 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:14.132224 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.133073 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.134016 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.134249 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.134307 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.136823 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:14.136824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:14.136825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:14.136833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:14.138232 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.138650 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.138651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.138651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.138711 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.138959 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.138960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.138960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.139076 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:14.139159 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.139429 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.139538 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:12:14.139776 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.130509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:50567)
0810 21:12:14.130665 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.130705 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.131164 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.131417 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.131420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.131950 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:14.132128 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.132132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.133166 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.133167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.133938 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.133941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.134343 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.134344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.134490 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.135011 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:14.135022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:14.136709 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:14.138335 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:14.138343 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:14.138345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:14.138574 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:14.138784 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:14.138785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:14.138785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:14.138893 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:14.139150 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:14.139158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:14.139293 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.139393 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.139548 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260810 21:12:14.140208 21477 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40667
I20260810 21:12:14.140316 21477 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:14.140491 21477 heartbeater.cc:511] Master 127.27.117.126:40667 requested a full tablet report, sending...
I20260810 21:12:14.140900 20905 ts_manager.cc:194] Registered new tserver with Master: 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851)
I20260810 21:12:14.141283 20905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:50567
W20260810 21:12:14.215497 21482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:14.215723 21482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:14.215758 21482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:14.216907 21482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:14.216961 21482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:14.217010 21482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:14.218420 21482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:46407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=35547
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:14.218675 21482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:14.218909 21482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:14.223662 21492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.223672 21490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.223728 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.223953 21482 server_base.cc:1034] running on GCE node
I20260810 21:12:14.224181 21482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:14.224427 21482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:14.225587 21482 hybrid_clock.cc:648] HybridClock initialized: now 1786396334225584 us; error 33 us; skew 500 ppm
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:14.228037 21482 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:14.228401 21482 webserver.cc:460] Webserver started at http://127.27.117.67:35547/ using document root <none> and password file <none>
I20260810 21:12:14.228592 21482 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:14.228643 21482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:14.229940 21482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:14.230484 21505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.230652 21482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:14.230741 21482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "b49afa89e4d944f2a74c5953f320dd7d"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "dace31efac2bdf586c01e32b7828e3cc"
server_key_iv: "9cb5d52e7cdb4871084b8253b4882a7c"
server_key_version: "encryptionkey@0"
I20260810 21:12:14.231031 21482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:14.248138 21482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:14.250146 21482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:14.250289 21482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:14.250550 21482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:14.250873 21482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:14.250909 21482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.250950 21482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:14.250974 21482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.258536 21482 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:46407
I20260810 21:12:14.258575 21706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:46407 every 8 connection(s)
I20260810 21:12:14.258932 21482 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:14.267575 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.259684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:38511 (local address 127.27.117.126:40667)
0810 21:12:14.259732 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.259734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.259946 (+ 212us) server_negotiation.cc:407] Connection header received
0810 21:12:14.260422 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.260423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.260433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.260484 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:14.261481 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.261967 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.262894 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.263081 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.263134 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.265182 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:14.265183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:14.265183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:14.265190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:14.266443 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.266808 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.266809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.266809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.266861 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.267067 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.267068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.267068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.267165 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:14.267216 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.267433 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.267510 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:12:14.267728 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 21482
I20260810 21:12:14.267827 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.259670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:38511)
0810 21:12:14.259828 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.259864 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.260338 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.260541 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.260544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.261210 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:14.261403 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.261408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.262045 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.262046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.262820 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.262823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.263154 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.263155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.263255 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.263694 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:14.263702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:14.265084 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:14.266517 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:14.266524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:14.266526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:14.266742 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:14.266915 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:14.266916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:14.266916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:14.267014 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:14.267217 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:14.267221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:14.267328 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.267408 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.267580 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260810 21:12:14.268607 21708 heartbeater.cc:347] Connected to a master server at 127.27.117.126:40667
I20260810 21:12:14.268700 21708 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:14.268910 21708 heartbeater.cc:511] Master 127.27.117.126:40667 requested a full tablet report, sending...
I20260810 21:12:14.269371 20905 ts_manager.cc:194] Registered new tserver with Master: b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
I20260810 21:12:14.269830 20905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:38511
I20260810 21:12:14.274933 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.268140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.27.117.126:40667)
0810 21:12:14.268174 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.268175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.268369 (+ 194us) server_negotiation.cc:407] Connection header received
0810 21:12:14.268447 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.268447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.268458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.268520 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:14.269956 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.270470 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.271239 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.271442 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.271514 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.272690 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:14.272691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:14.272692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:14.272699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:14.273850 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.274178 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.274179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.274180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.274235 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:14.274438 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:14.274439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:14.274439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:14.274545 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:14.274728 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.274766 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.274836 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 21:12:14.275630 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:12:14.278072 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.275982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.27.117.126:40667)
0810 21:12:14.276043 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.276045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.276121 (+ 76us) server_negotiation.cc:407] Connection header received
0810 21:12:14.276171 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.276172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.276198 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.276259 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:14.276325 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.276744 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.277482 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.277652 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.277701 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.277817 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:14.277900 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:14.277954 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.277982 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.278002 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:12:14.278374 20905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:12:14.280000 20905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:12:14.282187 21107 tablet_service.cc:1467] Processing CreateTablet for tablet 3a11a6acd46d44838ff608c776b9a626 (DEFAULT_TABLE table=test-table [id=975ab2bc23c84005930b98794e044640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:14.282405 21107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a11a6acd46d44838ff608c776b9a626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:14.284174 21038 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Bootstrap starting.
I20260810 21:12:14.284618 21038 tablet_bootstrap.cc:654] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:14.285617 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.281874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:41851 (local address 127.0.0.1:60546)
0810 21:12:14.281909 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.281921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.281964 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.282404 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.282406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.282408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:14.282458 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.282460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.283660 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.283661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.284514 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.284517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.285390 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.285391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.285451 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.285454 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.285510 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.285554 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 21:12:14.285643 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.282031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.27.117.66:41851)
0810 21:12:14.282244 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.282249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.282261 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:12:14.282302 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.282304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.282318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.282450 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:14.282540 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.283559 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.284642 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.285302 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.285391 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.285451 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.285527 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.285567 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":125,"threads_started":1}
I20260810 21:12:14.286048 21038 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: No bootstrap required, opened a new log
I20260810 21:12:14.286115 21038 ts_tablet_manager.cc:1397] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:12:14.286275 21038 raft_consensus.cc:348] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.286350 21038 raft_consensus.cc:374] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:14.286402 21038 raft_consensus.cc:729] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092430078ca5431e89bca6157a32ba5a, State: Initialized, Role: FOLLOWER
I20260810 21:12:14.286449 21038 consensus_queue.cc:260] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.286405 21348 tablet_service.cc:1467] Processing CreateTablet for tablet 3a11a6acd46d44838ff608c776b9a626 (DEFAULT_TABLE table=test-table [id=975ab2bc23c84005930b98794e044640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:14.286557 21038 ts_tablet_manager.cc:1428] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:14.286739 21348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a11a6acd46d44838ff608c776b9a626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:14.287901 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.282146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.27.117.67:46407)
0810 21:12:14.283744 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.283748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.284358 (+ 610us) server_negotiation.cc:407] Connection header received
0810 21:12:14.284395 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.284397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.284409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.284492 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:14.284691 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.285548 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.286941 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.287521 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.287580 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.287613 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.287782 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.287827 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":70,"threads_started":1}
I20260810 21:12:14.288096 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.281935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46407 (local address 127.0.0.1:33148)
0810 21:12:14.284226 (+ 2291us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.284238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.284294 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.284497 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.284498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.284500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:14.284547 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.284549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.285653 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.285654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.286858 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.286861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.287602 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.287604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.287667 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.287671 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.287980 (+ 309us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.288028 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":74,"threads_started":1}
I20260810 21:12:14.288846 21591 tablet_service.cc:1467] Processing CreateTablet for tablet 3a11a6acd46d44838ff608c776b9a626 (DEFAULT_TABLE table=test-table [id=975ab2bc23c84005930b98794e044640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:14.289191 21591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a11a6acd46d44838ff608c776b9a626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:14.290460 21724 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Bootstrap starting.
I20260810 21:12:14.291050 21724 tablet_bootstrap.cc:654] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:14.291538 21724 log.cc:824] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:14.292277 21724 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: No bootstrap required, opened a new log
I20260810 21:12:14.292356 21724 ts_tablet_manager.cc:1397] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:12:14.292380 21726 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Bootstrap starting.
I20260810 21:12:14.293064 21726 tablet_bootstrap.cc:654] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:14.293648 21726 log.cc:824] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:14.294410 21726 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: No bootstrap required, opened a new log
I20260810 21:12:14.294485 21726 ts_tablet_manager.cc:1397] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:14.294561 21724 raft_consensus.cc:348] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.294716 21724 raft_consensus.cc:374] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:14.294759 21724 raft_consensus.cc:729] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98fda818aae54e15a0d351ae5eb38a75, State: Initialized, Role: FOLLOWER
I20260810 21:12:14.294878 21724 consensus_queue.cc:260] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.295166 21724 ts_tablet_manager.cc:1428] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:12:14.295176 21477 heartbeater.cc:503] Master 127.27.117.126:40667 was elected leader, sending a full tablet report...
I20260810 21:12:14.296089 21726 raft_consensus.cc:348] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.296206 21726 raft_consensus.cc:374] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:14.296226 21726 raft_consensus.cc:729] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49afa89e4d944f2a74c5953f320dd7d, State: Initialized, Role: FOLLOWER
I20260810 21:12:14.296360 21726 consensus_queue.cc:260] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.296618 21726 ts_tablet_manager.cc:1428] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:14.296655 21708 heartbeater.cc:503] Master 127.27.117.126:40667 was elected leader, sending a full tablet report...
W20260810 21:12:14.380868 21480 tablet.cc:2369] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:12:14.443773 21238 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:14.443928 21238 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.444269 21238 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
I20260810 21:12:14.448288 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.444550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:54461 (local address 127.27.117.66:41851)
0810 21:12:14.444609 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.444611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.444622 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:12:14.444655 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.444656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.444667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.444734 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:14.444962 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.445716 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.447260 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.447855 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.447936 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.447944 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.448155 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.448204 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:12:14.448274 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.444646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:44055 (local address 127.27.117.67:46407)
0810 21:12:14.444694 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.444696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.444798 (+ 102us) server_negotiation.cc:407] Connection header received
0810 21:12:14.444861 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.444862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.444869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.444906 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:14.445082 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.445755 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.447261 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.447861 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.447936 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.447944 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.448155 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.448204 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:12:14.448259 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.444457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:41851 (local address 127.27.117.65:54461)
0810 21:12:14.444538 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.444549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.444596 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.444758 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.444759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.444762 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:14.444866 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.444873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.445814 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.445815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.447155 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.447159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.447943 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.447945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.448052 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.448055 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.448107 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.448156 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":57}
I20260810 21:12:14.448271 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.444510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46407 (local address 127.27.117.65:44055)
0810 21:12:14.444697 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.444707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.444772 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.444933 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:14.444934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:14.444935 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:14.445015 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.445016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.445825 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.445825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.447155 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:14.447159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.447960 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.447960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:14.448040 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.448044 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:14.448107 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 21:12:14.448154 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":86,"thread_start_us":110,"threads_started":1}
I20260810 21:12:14.448701 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11a6acd46d44838ff608c776b9a626" candidate_uuid: "092430078ca5431e89bca6157a32ba5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98fda818aae54e15a0d351ae5eb38a75" is_pre_election: true
I20260810 21:12:14.448706 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11a6acd46d44838ff608c776b9a626" candidate_uuid: "092430078ca5431e89bca6157a32ba5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49afa89e4d944f2a74c5953f320dd7d" is_pre_election: true
I20260810 21:12:14.448891 21630 raft_consensus.cc:2454] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092430078ca5431e89bca6157a32ba5a in term 0.
I20260810 21:12:14.448894 21388 raft_consensus.cc:2454] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092430078ca5431e89bca6157a32ba5a in term 0.
I20260810 21:12:14.449152 21033 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a, 98fda818aae54e15a0d351ae5eb38a75; no voters:
I20260810 21:12:14.449295 21238 raft_consensus.cc:2781] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:12:14.449355 21238 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:12:14.449373 21238 raft_consensus.cc:3037] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:14.450076 21238 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.450220 21238 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 1 election: Requested vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
I20260810 21:12:14.450372 21388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11a6acd46d44838ff608c776b9a626" candidate_uuid: "092430078ca5431e89bca6157a32ba5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98fda818aae54e15a0d351ae5eb38a75"
I20260810 21:12:14.450373 21630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11a6acd46d44838ff608c776b9a626" candidate_uuid: "092430078ca5431e89bca6157a32ba5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b49afa89e4d944f2a74c5953f320dd7d"
I20260810 21:12:14.450451 21388 raft_consensus.cc:3037] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:14.450470 21630 raft_consensus.cc:3037] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:14.451165 21388 raft_consensus.cc:2454] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092430078ca5431e89bca6157a32ba5a in term 1.
I20260810 21:12:14.451275 21630 raft_consensus.cc:2454] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092430078ca5431e89bca6157a32ba5a in term 1.
I20260810 21:12:14.451328 21033 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a, 98fda818aae54e15a0d351ae5eb38a75; no voters:
I20260810 21:12:14.451489 21238 raft_consensus.cc:2781] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:14.451629 21238 raft_consensus.cc:686] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 LEADER]: Becoming Leader. State: Replica: 092430078ca5431e89bca6157a32ba5a, State: Running, Role: LEADER
I20260810 21:12:14.451684 21238 consensus_queue.cc:237] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.452445 20905 catalog_manager.cc:5495] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 092430078ca5431e89bca6157a32ba5a (127.27.117.65). New cstate: current_term: 1 leader_uuid: "092430078ca5431e89bca6157a32ba5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } health_report { overall_health: HEALTHY } } }
I20260810 21:12:14.465746 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.463674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.27.117.65:37859)
0810 21:12:14.463722 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:14.463724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:14.463756 (+ 32us) server_negotiation.cc:407] Connection header received
0810 21:12:14.463783 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:14.463783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:14.463797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:14.463850 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:14.463950 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.464380 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.465088 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:14.465266 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:14.465321 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:14.465504 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:14.465591 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:14.465640 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:14.465681 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 21:12:14.465698 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:12:14.467196 21388 raft_consensus.cc:1261] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Refusing update from remote peer 092430078ca5431e89bca6157a32ba5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:14.467197 21630 raft_consensus.cc:1261] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Refusing update from remote peer 092430078ca5431e89bca6157a32ba5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:14.467519 21741 consensus_queue.cc:1035] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:14.467629 21238 consensus_queue.cc:1035] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:14.470865 21750 mvcc.cc:205] Tried to move back new op lower bound from 7317079385975205888 to 7317079385914646528. Current Snapshot: MvccSnapshot[applied={T|T < 7317079385975205888}]
I20260810 21:12:14.472256 20903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:14.472358 20903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:14.473069 20903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Sending DeleteTablet for 3 replicas of tablet 3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:14.473402 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 20968
I20260810 21:12:14.473400 21591 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a11a6acd46d44838ff608c776b9a626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:14 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33148
I20260810 21:12:14.480290 21348 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a11a6acd46d44838ff608c776b9a626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:14 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60546
I20260810 21:12:14.480515 21763 tablet_replica.cc:330] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: stopping tablet replica
I20260810 21:12:14.480615 21763 raft_consensus.cc:2229] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:14.480718 21763 raft_consensus.cc:2258] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:14.481225 21763 ts_tablet_manager.cc:1905] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:14.483004 21763 ts_tablet_manager.cc:1918] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:14.483078 21763 log.cc:1196] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:14.483319 21763 ts_tablet_manager.cc:1939] T 3a11a6acd46d44838ff608c776b9a626 P 98fda818aae54e15a0d351ae5eb38a75: Deleting consensus metadata
I20260810 21:12:14.483721 20877 catalog_manager.cc:4841] TS 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): tablet 3a11a6acd46d44838ff608c776b9a626 (table test-table [id=975ab2bc23c84005930b98794e044640]) successfully deleted
I20260810 21:12:14.484640 21762 tablet_replica.cc:330] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: stopping tablet replica
I20260810 21:12:14.484745 21762 raft_consensus.cc:2229] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:14.484865 21762 raft_consensus.cc:2258] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:14.486670 21762 ts_tablet_manager.cc:1905] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:14.488595 21762 ts_tablet_manager.cc:1918] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:14.488692 21762 log.cc:1196] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:14.489032 21762 ts_tablet_manager.cc:1939] T 3a11a6acd46d44838ff608c776b9a626 P b49afa89e4d944f2a74c5953f320dd7d: Deleting consensus metadata
I20260810 21:12:14.489432 20873 catalog_manager.cc:4841] TS b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): tablet 3a11a6acd46d44838ff608c776b9a626 (table test-table [id=975ab2bc23c84005930b98794e044640]) successfully deleted
W20260810 21:12:14.491475 20879 connection.cc:535] client connection to 127.27.117.65:37859 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 21:12:14.491573 20879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 21:12:14.491935 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 21237
I20260810 21:12:14.492038 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.491904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37859 (local address 127.0.0.1:51794)
0810 21:12:14.491953 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.491992 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 21:12:14.502942 20879 catalog_manager.cc:4584] TS 092430078ca5431e89bca6157a32ba5a (127.27.117.65:37859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a11a6acd46d44838ff608c776b9a626: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37859: connect: Connection refused (error 111)
I20260810 21:12:14.509049 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 21482
I20260810 21:12:14.522743 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 20842
I20260810 21:12:14.533100 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40667
--webserver_interface=127.27.117.126
--webserver_port=39043
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:14.612649 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:14.612890 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:14.612933 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:14.614053 21765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:14.614109 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:14.614136 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:14.614166 21765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:14.614189 21765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:14.615599 21765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=39043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:14.615856 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:14.616060 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:14.620740 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.620752 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.620940 21765 server_base.cc:1034] running on GCE node
W20260810 21:12:14.620795 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.621296 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:14.621584 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:14.622745 21765 hybrid_clock.cc:648] HybridClock initialized: now 1786396334622732 us; error 40 us; skew 500 ppm
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:14.625156 21765 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:14.625526 21765 webserver.cc:460] Webserver started at http://127.27.117.126:39043/ using document root <none> and password file <none>
I20260810 21:12:14.625697 21765 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:14.625736 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:14.626926 21765 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:14.627532 21789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.627715 21765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.627799 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "362d4cf2d0ee4b47b533584e877c8316"
format_stamp: "Formatted at 2026-08-10 21:12:12 on dist-test-slave-2kcd"
server_key: "851110480fabb653566bf4f965362414"
server_key_iv: "f01732ac6c1081cd9d4519dd2922459b"
server_key_version: "encryptionkey@0"
I20260810 21:12:14.628082 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:14.656714 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:14.658902 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:14.659011 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:14.662885 21765 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:40667
I20260810 21:12:14.662968 21871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:40667 every 8 connection(s)
I20260810 21:12:14.663396 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:14.665544 21872 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:12:14.666208 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap starting.
I20260810 21:12:14.668103 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 21765
I20260810 21:12:14.668427 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:37859
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:40667
--webserver_port=36621
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:14.676100 21872 log.cc:824] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:14.678488 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:14.678813 21872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316: Bootstrap complete.
I20260810 21:12:14.681264 21872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:14.681519 21872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362d4cf2d0ee4b47b533584e877c8316, State: Initialized, Role: FOLLOWER
I20260810 21:12:14.681704 21872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:14.681787 21872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:14.681838 21872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:14.681888 21872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 2 FOLLOWER]: Advancing to term 3
I20260810 21:12:14.683007 21872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:14.683126 21872 leader_election.cc:302] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362d4cf2d0ee4b47b533584e877c8316; no voters:
I20260810 21:12:14.683331 21872 leader_election.cc:288] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 21:12:14.683367 21881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 21:12:14.683483 21881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [term 3 LEADER]: Becoming Leader. State: Replica: 362d4cf2d0ee4b47b533584e877c8316, State: Running, Role: LEADER
I20260810 21:12:14.683581 21881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } }
I20260810 21:12:14.683609 21872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:14.684062 21882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "362d4cf2d0ee4b47b533584e877c8316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } } }
I20260810 21:12:14.684058 21881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362d4cf2d0ee4b47b533584e877c8316. Latest consensus state: current_term: 3 leader_uuid: "362d4cf2d0ee4b47b533584e877c8316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362d4cf2d0ee4b47b533584e877c8316" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 40667 } } }
I20260810 21:12:14.684175 21882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:14.684191 21881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362d4cf2d0ee4b47b533584e877c8316 [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:14.684412 21897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:14.685322 21897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:14.685427 21897 catalog_manager.cc:1259] Loaded cluster ID: 123b68b9dcf644a9971b605bae117cb0
I20260810 21:12:14.685453 21897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 21:12:14.685829 21897 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe6b1a329ad google::LogMessage::Fail() at ??:0
@ 0x7fe6b1a3588a google::LogMessage::SendToLog() at ??:0
@ 0x7fe6b1a324de google::LogMessage::Flush() at ??:0
@ 0x7fe6b1a33a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe6b99530f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe6b995c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe6b39000d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe6b38f5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe6b47596db start_thread at ??:0
@ 0x7fe6b053061f clone at ??:0
W20260810 21:12:14.759071 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:14.759301 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:14.759342 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:14.760474 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:14.760546 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:14.760594 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:14.762141 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:37859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=36621
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:14.762421 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:14.762660 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:14.767752 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.767741 21937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.767755 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.768191 21876 server_base.cc:1034] running on GCE node
I20260810 21:12:14.768406 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:14.768656 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:14.769814 21876 hybrid_clock.cc:648] HybridClock initialized: now 1786396334769801 us; error 33 us; skew 500 ppm
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:14.772441 21876 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:14.772982 21876 webserver.cc:460] Webserver started at http://127.27.117.65:36621/ using document root <none> and password file <none>
I20260810 21:12:14.773227 21876 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:14.773294 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:14.774685 21876 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:12:14.775497 21960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.775650 21876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:14.775738 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "092430078ca5431e89bca6157a32ba5a"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "b3b0abeb8434190d5bf0fed2620836ab"
server_key_iv: "5f96013bd85f647c00c5b5b6b465e825"
server_key_version: "encryptionkey@0"
I20260810 21:12:14.776080 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:14.790812 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:14.792936 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:14.793084 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:14.793305 21876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:14.793751 21971 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 21:12:14.795706 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 21:12:14.795763 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 21:12:14.795805 21876 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 21:12:14.796434 21876 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 21:12:14.796487 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 21:12:14.796561 21971 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Bootstrap starting.
I20260810 21:12:14.803968 21876 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37859
I20260810 21:12:14.804134 22157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37859 every 8 connection(s)
I20260810 21:12:14.804438 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:12:14.808182 21971 log.cc:824] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:14.810963 21971 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:14.811305 21971 tablet_bootstrap.cc:492] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Bootstrap complete.
I20260810 21:12:14.812750 21971 ts_tablet_manager.cc:1397] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260810 21:12:14.814404 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 21876
I20260810 21:12:14.814296 21971 raft_consensus.cc:348] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.814644 21971 raft_consensus.cc:729] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092430078ca5431e89bca6157a32ba5a, State: Initialized, Role: FOLLOWER
I20260810 21:12:14.814680 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:41851
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:40667
--webserver_port=46675
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:14.814800 21971 consensus_queue.cc:260] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:14.815193 21971 ts_tablet_manager.cc:1428] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260810 21:12:14.912653 22161 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:12:14.805150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:50531)
0810 21:12:14.805325 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.805368 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:14.805813 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:14.912402 (+106589us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
W20260810 21:12:14.912971 22158 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:40667 (0 consecutive failures): Network error: Failed to ping master at 127.27.117.126:40667: Client connection negotiation failed: client connection to 127.27.117.126:40667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 21:12:14.913327 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.913208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:50361)
0810 21:12:14.913261 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.913300 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 21:12:14.913709 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.913630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:60539)
0810 21:12:14.913670 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.913694 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 21:12:14.913836 22158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:12:14.929600 22165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:14.929809 22165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:14.929847 22165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:14.930965 22165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:14.931032 22165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:14.931080 22165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:14.932571 22165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:41851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=46675
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:14.932840 22165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:14.933066 22165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:14.937867 22178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.937896 22165 server_base.cc:1034] running on GCE node
W20260810 21:12:14.937853 22180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:14.937842 22182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:14.938206 22165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:14.938460 22165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:14.939600 22165 hybrid_clock.cc:648] HybridClock initialized: now 1786396334939587 us; error 28 us; skew 500 ppm
Aug 10 21:12:14 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:14.942118 22165 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:14.942529 22165 webserver.cc:460] Webserver started at http://127.27.117.66:46675/ using document root <none> and password file <none>
I20260810 21:12:14.942740 22165 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:14.942785 22165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:14.944072 22165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:14.944639 22191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.944819 22165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:14.944947 22165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "98fda818aae54e15a0d351ae5eb38a75"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "1d1b1ae8e1d7a0422be70458efa60fe2"
server_key_iv: "0e6f6eddbe3fe7ecd30448f4b2bb5582"
server_key_version: "encryptionkey@0"
I20260810 21:12:14.945266 22165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:14.971680 22165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:14.973469 22165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:14.973603 22165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:14.973786 22165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:14.974045 22165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:14.974077 22165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.974099 22165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:14.974112 22165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:14.980111 22165 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:41851
I20260810 21:12:14.980190 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:41851 every 8 connection(s)
I20260810 21:12:14.980516 22165 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:12:14.981549 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.981167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:59333)
0810 21:12:14.981326 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.981401 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260810 21:12:14.981972 22389 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:40667 (0 consecutive failures): Network error: Failed to ping master at 127.27.117.126:40667: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
I20260810 21:12:14.982278 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.982184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:45283)
0810 21:12:14.982229 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.982259 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 21:12:14.982530 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 22165
I20260810 21:12:14.982808 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:46407
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:40667
--webserver_port=35547
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:14.984848 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:14.984721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:34227)
0810 21:12:14.984766 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:14.984830 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 21:12:14.984956 22389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:12:15.064919 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:15.065125 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:15.065162 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:15.066228 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:15.066289 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:15.066334 22394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:15.067798 22394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:46407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=35547
--tserver_master_addrs=127.27.117.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:15.068064 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:15.068282 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:15.073046 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:15.073066 22408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:15.073058 22410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:15.073357 22394 server_base.cc:1034] running on GCE node
I20260810 21:12:15.073560 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:15.073829 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:15.074985 22394 hybrid_clock.cc:648] HybridClock initialized: now 1786396335074976 us; error 29 us; skew 500 ppm
Aug 10 21:12:15 dist-test-slave-2kcd krb5kdc[19862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:15.077421 22394 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:15.077807 22394 webserver.cc:460] Webserver started at http://127.27.117.67:35547/ using document root <none> and password file <none>
I20260810 21:12:15.078042 22394 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:15.078136 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:15.079450 22394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:15.080178 22429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:15.080331 22394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:15.080435 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "b49afa89e4d944f2a74c5953f320dd7d"
format_stamp: "Formatted at 2026-08-10 21:12:13 on dist-test-slave-2kcd"
server_key: "dace31efac2bdf586c01e32b7828e3cc"
server_key_iv: "9cb5d52e7cdb4871084b8253b4882a7c"
server_key_version: "encryptionkey@0"
I20260810 21:12:15.080766 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:15.101744 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:15.103564 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:15.103718 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:15.103951 22394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:15.104264 22394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:15.104316 22394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:15.104357 22394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:15.104389 22394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:15.110441 22394 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:46407
I20260810 21:12:15.110497 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:46407 every 8 connection(s)
I20260810 21:12:15.110827 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:12:15.112836 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:15.111516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:48381)
0810 21:12:15.112593 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:15.112686 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":129,"threads_started":1}
W20260810 21:12:15.113250 22629 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:40667 (0 consecutive failures): Network error: Failed to ping master at 127.27.117.126:40667: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
I20260810 21:12:15.113540 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:15.113443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:33979)
0810 21:12:15.113488 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:15.113519 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 21:12:15.113831 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:15.113759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:47197)
0810 21:12:15.113794 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:15.113817 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 21:12:15.113956 22629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 21:12:15.118634 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 22394
I20260810 21:12:15.914379 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:15.914146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:37877)
0810 21:12:15.914275 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:15.914310 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260810 21:12:15.985428 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:15.985235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:57473)
0810 21:12:15.985348 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:15.985382 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260810 21:12:16.114524 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.114325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:54063)
0810 21:12:16.114437 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:16.114472 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260810 21:12:16.224970 22643 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:16.225133 22643 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:16.225507 22643 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
Aug 10 21:12:16 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.67@KRBTEST.COM
Aug 10 21:12:16 dist-test-slave-2kcd krb5kdc[19862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396334, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.66@KRBTEST.COM
I20260810 21:12:16.235672 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.225762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:58265 (local address 127.27.117.66:41851)
0810 21:12:16.225968 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:16.225979 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:16.226017 (+ 38us) server_negotiation.cc:407] Connection header received
0810 21:12:16.226452 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:16.226454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:16.226493 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:16.226823 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:16.227344 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.228192 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.229333 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.229710 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.229842 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:16.232452 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:16.232456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:16.232456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:16.232472 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:16.234078 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234484 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:16.234487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:16.234487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:16.234576 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234833 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:16.234834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:16.234835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:16.235067 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:16.235164 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:16.235432 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 21:12:16.235556 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260810 21:12:16.235659 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.225666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:46407 (local address 127.27.117.65:60225)
0810 21:12:16.225819 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:16.225830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:16.225908 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:16.226724 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:16.226727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:16.227404 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:16.227441 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:16.227443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.228606 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.228607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:16.229239 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:16.229241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.229819 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.229820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:16.229934 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:16.230496 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:16.230505 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:16.232026 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:16.234064 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:16.234074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:16.234398 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:16.234651 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:16.234652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:16.234763 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:16.235134 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:16.235141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:16.235270 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 21:12:16.235371 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 21:12:16.235563 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260810 21:12:16.235697 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.225823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:60225 (local address 127.27.117.67:46407)
0810 21:12:16.225987 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:16.226004 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:16.226031 (+ 27us) server_negotiation.cc:407] Connection header received
0810 21:12:16.226452 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:16.226454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:16.226545 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:16.226702 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:16.227551 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.228501 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.229327 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.229710 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.229828 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:16.232139 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:16.232143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:16.232143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:16.232155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:16.233979 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234480 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:16.234483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:16.234483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:16.234580 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234830 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:16.234831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:16.234832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:16.235061 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:16.235145 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:16.235443 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 21:12:16.235578 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260810 21:12:16.235777 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.225640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:41851 (local address 127.27.117.65:58265)
0810 21:12:16.225755 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:16.225767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:16.225836 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:16.226725 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:16.226728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:16.227058 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:16.227244 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:16.227250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.228293 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.228294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:16.229237 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:16.229240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:16.229821 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:16.229822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:16.229935 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:16.230562 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:16.230567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:16.232335 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:16.234164 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:16.234165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:16.234400 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:16.234657 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:16.234658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:16.234658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:16.234766 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:16.235157 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:16.235158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:16.235269 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 21:12:16.235373 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 21:12:16.235645 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":139}
W20260810 21:12:16.236873 21961 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
W20260810 21:12:16.236968 21965 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:16.237035 21965 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a; no voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d
I20260810 21:12:16.237191 22643 raft_consensus.cc:2726] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:12:16.915271 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.914987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:51475)
0810 21:12:16.915137 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:16.915205 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260810 21:12:16.986006 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:16.985812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:53011)
0810 21:12:16.985928 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:16.985961 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260810 21:12:17.115154 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:17.114947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:60125)
0810 21:12:17.115072 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:17.115111 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260810 21:12:17.915982 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:17.915741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:55753)
0810 21:12:17.915896 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:17.915933 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260810 21:12:17.986617 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:17.986401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:52283)
0810 21:12:17.986548 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:17.986579 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260810 21:12:18.115862 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:18.115667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:45165)
0810 21:12:18.115791 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:18.115826 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260810 21:12:18.403601 22660 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:18.403731 22660 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:18.403929 22660 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
W20260810 21:12:18.404343 21965 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
W20260810 21:12:18.404438 21961 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:18.404536 21961 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a; no voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d
I20260810 21:12:18.404610 22660 raft_consensus.cc:2726] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:12:18.916848 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:18.916461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:55753)
0810 21:12:18.916663 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:18.916700 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":64,"threads_started":1}
I20260810 21:12:18.987248 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:18.987063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:43557)
0810 21:12:18.987179 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:18.987208 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260810 21:12:19.116500 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:19.116290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:54367)
0810 21:12:19.116413 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:19.116452 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260810 21:12:19.917574 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:19.917373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:46717)
0810 21:12:19.917491 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:19.917526 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260810 21:12:19.987856 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:19.987649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:48727)
0810 21:12:19.987784 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:19.987819 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260810 21:12:20.117270 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:20.117065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:36073)
0810 21:12:20.117199 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:20.117234 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260810 21:12:20.918197 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:20.917975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:41415)
0810 21:12:20.918115 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:20.918148 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260810 21:12:20.988749 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:20.988509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:41383)
0810 21:12:20.988680 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:20.988713 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
I20260810 21:12:21.066746 22671 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:21.066844 22671 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:21.067080 22671 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
W20260810 21:12:21.067452 21965 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
W20260810 21:12:21.067528 21961 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:21.067584 21961 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a; no voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d
I20260810 21:12:21.067649 22671 raft_consensus.cc:2726] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:12:21.117894 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:21.117692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:37053)
0810 21:12:21.117823 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:21.117855 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260810 21:12:21.918991 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:21.918752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:58383)
0810 21:12:21.918906 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:21.918942 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260810 21:12:21.989364 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:21.989146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:55291)
0810 21:12:21.989292 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:21.989327 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260810 21:12:22.118746 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:22.118510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:51863)
0810 21:12:22.118668 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:22.118706 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260810 21:12:22.919749 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:22.919420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:54893)
0810 21:12:22.919662 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:22.919699 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":70,"threads_started":1}
I20260810 21:12:22.990039 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:22.989799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:34067)
0810 21:12:22.989954 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:22.989995 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260810 21:12:23.119503 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:23.119285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:55215)
0810 21:12:23.119424 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:23.119468 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260810 21:12:23.920534 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:23.920303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:57561)
0810 21:12:23.920436 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:23.920474 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260810 21:12:23.990693 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:23.990494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:45061)
0810 21:12:23.990616 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:23.990652 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260810 21:12:24.120323 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:24.120119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:47889)
0810 21:12:24.120249 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:24.120285 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260810 21:12:24.921311 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:24.921070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:41829)
0810 21:12:24.921209 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:24.921247 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260810 21:12:24.991271 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:24.991081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:60979)
0810 21:12:24.991196 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:24.991231 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260810 21:12:25.121131 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:25.120910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:39299)
0810 21:12:25.121054 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:25.121091 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260810 21:12:25.715667 22701 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:25.715766 22701 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:25.715974 22701 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
W20260810 21:12:25.716377 21965 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
W20260810 21:12:25.716456 21961 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:25.716506 21961 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a; no voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d
I20260810 21:12:25.716583 22701 raft_consensus.cc:2726] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:12:25.921976 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:25.921763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:35361)
0810 21:12:25.921899 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:25.921930 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260810 21:12:25.992007 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:25.991759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:47241)
0810 21:12:25.991914 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:25.991951 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260810 21:12:26.121793 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:26.121590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:36451)
0810 21:12:26.121712 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:26.121742 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260810 21:12:26.922658 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:26.922435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:44993)
0810 21:12:26.922566 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:26.922606 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260810 21:12:26.992718 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:26.992480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:47763)
0810 21:12:26.992642 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:26.992675 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260810 21:12:27.122512 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:27.122257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:54821)
0810 21:12:27.122424 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:27.122464 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260810 21:12:27.923377 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:27.923134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:59291)
0810 21:12:27.923294 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:27.923325 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":108,"threads_started":1}
I20260810 21:12:27.946983 22716 raft_consensus.cc:482] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:27.947108 22716 raft_consensus.cc:504] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98fda818aae54e15a0d351ae5eb38a75" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41851 } } peers { permanent_uuid: "b49afa89e4d944f2a74c5953f320dd7d" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 46407 } } peers { permanent_uuid: "092430078ca5431e89bca6157a32ba5a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37859 } }
I20260810 21:12:27.947324 22716 leader_election.cc:288] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851), b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407)
W20260810 21:12:27.947782 21961 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b49afa89e4d944f2a74c5953f320dd7d (127.27.117.67:46407): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
W20260810 21:12:27.947880 21965 leader_election.cc:341] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 98fda818aae54e15a0d351ae5eb38a75 (127.27.117.66:41851): Not found: Tablet not found: 3a11a6acd46d44838ff608c776b9a626
I20260810 21:12:27.947923 21965 leader_election.cc:302] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 092430078ca5431e89bca6157a32ba5a; no voters: 98fda818aae54e15a0d351ae5eb38a75, b49afa89e4d944f2a74c5953f320dd7d
I20260810 21:12:27.948012 22716 raft_consensus.cc:2726] T 3a11a6acd46d44838ff608c776b9a626 P 092430078ca5431e89bca6157a32ba5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:12:27.993438 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:27.993238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:39367)
0810 21:12:27.993365 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:27.993397 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260810 21:12:28.123168 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:28.122973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:40327)
0810 21:12:28.123095 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:28.123130 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260810 21:12:28.924161 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:28.923906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:44687)
0810 21:12:28.924071 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:28.924113 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260810 21:12:28.994169 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:28.993941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:35795)
0810 21:12:28.994077 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:28.994123 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260810 21:12:29.123781 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:29.123581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:37139)
0810 21:12:29.123706 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:29.123741 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260810 21:12:29.924970 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:29.924686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.65:56857)
0810 21:12:29.924881 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:29.924918 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
I20260810 21:12:29.994837 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:29.994658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.66:52387)
0810 21:12:29.994770 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:29.994800 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260810 21:12:30.124478 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:30.124283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:40667 (local address 127.27.117.67:41175)
0810 21:12:30.124400 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:30.124433 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:40667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260810 21:12:30.128530 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 21876
I20260810 21:12:30.147495 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 22165
I20260810 21:12:30.157980 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 22394
2026-08-10T21:12:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18770 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:12:30 dist-test-slave-2kcd krb5kdc[22742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:12:30 dist-test-slave-2kcd krb5kdc[22742](info): set up 2 sockets
Aug 10 21:12:30 dist-test-slave-2kcd krb5kdc[22742](info): commencing operation
krb5kdc: starting...
W20260810 21:12:31.449151 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:12:31 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:12:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:12:31.573704 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45155
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:31.657961 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:31.658212 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:31.658253 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:31.659377 22763 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 21:12:31.659451 22763 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 21:12:31.659475 22763 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 21:12:31.659497 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:31.659534 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:31.659569 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:31.659600 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:31.659631 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:31.661160 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:31.661453 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:31.661667 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:31.666987 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:31.667016 22763 server_base.cc:1034] running on GCE node
W20260810 21:12:31.667011 22773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:31.666965 22772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:31.667554 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:31.667883 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:31.669061 22763 hybrid_clock.cc:648] HybridClock initialized: now 1786396351669046 us; error 33 us; skew 500 ppm
Aug 10 21:12:31 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:31.671592 22763 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:31.672119 22763 webserver.cc:460] Webserver started at http://127.27.117.126:34061/ using document root <none> and password file <none>
I20260810 21:12:31.672374 22763 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:31.672449 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:31.672606 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:31.673748 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "c1645a25315c47f1b501a78fe2a2ec6f"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "6d0e1713615dcec612b57cdf97e6871b"
server_key_iv: "4ee9f57855b1b204825a024ee274ef5a"
server_key_version: "encryptionkey@0"
I20260810 21:12:31.674168 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "c1645a25315c47f1b501a78fe2a2ec6f"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "6d0e1713615dcec612b57cdf97e6871b"
server_key_iv: "4ee9f57855b1b204825a024ee274ef5a"
server_key_version: "encryptionkey@0"
I20260810 21:12:31.675588 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:12:31.676322 22789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:31.676477 22763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:31.676582 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "c1645a25315c47f1b501a78fe2a2ec6f"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "6d0e1713615dcec612b57cdf97e6871b"
server_key_iv: "4ee9f57855b1b204825a024ee274ef5a"
server_key_version: "encryptionkey@0"
I20260810 21:12:31.676667 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:31.688138 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:31.689905 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:31.690076 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:31.694296 22763 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:45155
I20260810 21:12:31.694332 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:45155 every 8 connection(s)
I20260810 21:12:31.694680 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:31.695533 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:31.697863 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap starting.
I20260810 21:12:31.698480 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 22763
I20260810 21:12:31.698573 22886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:31.698596 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:12:31.698803 28117 external_mini_cluster.cc:1428] Setting key 47243d394b77e4ec389f56f5bdccad31
I20260810 21:12:31.698925 22886 log.cc:824] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:31.699694 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: No bootstrap required, opened a new log
I20260810 21:12:31.702116 22886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:31.702288 22886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:31.702327 22886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1645a25315c47f1b501a78fe2a2ec6f, State: Initialized, Role: FOLLOWER
I20260810 21:12:31.702514 22886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:31.702600 22886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:31.702625 22886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:31.702675 22886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:31.703299 22886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:31.703404 22886 leader_election.cc:302] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1645a25315c47f1b501a78fe2a2ec6f; no voters:
I20260810 21:12:31.703570 22886 leader_election.cc:288] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:12:31.703639 22893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:31.703815 22886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:31.703837 22893 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 LEADER]: Becoming Leader. State: Replica: c1645a25315c47f1b501a78fe2a2ec6f, State: Running, Role: LEADER
I20260810 21:12:31.703928 22893 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:31.704296 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1645a25315c47f1b501a78fe2a2ec6f. Latest consensus state: current_term: 1 leader_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } } }
I20260810 21:12:31.704380 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } } }
I20260810 21:12:31.704497 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:31.704434 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:31.704756 22918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:31.705322 22918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:31.706818 22918 catalog_manager.cc:1347] Generated new cluster ID: 22555892b2dc4d9f87f29f4690311fa0
I20260810 21:12:31.706887 22918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:12:31 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:31.712930 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:31.699296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.27.117.126:45155)
0810 21:12:31.699475 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:31.699483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:31.700942 (+ 1459us) server_negotiation.cc:407] Connection header received
0810 21:12:31.702066 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:31.702075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:31.702137 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:31.702295 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:31.703071 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:31.704484 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:31.705736 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:31.706206 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:31.706328 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:31.709126 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:31.709130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:31.709131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:31.709142 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:31.711116 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:31.711516 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:31.711519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:31.711519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:31.711639 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:31.711851 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:31.711852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:31.711852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:31.712175 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:31.712317 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:31.712425 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 21:12:31.712571 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260810 21:12:31.721158 22918 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:12:31.721819 22918 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:31.728240 22918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Generated new TSK 0
I20260810 21:12:31.728420 22918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:12:31.755468 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45155
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:31.836601 22944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:31.836830 22944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:31.836872 22944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:31.837934 22944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:31.837975 22944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:31.838006 22944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:31.839373 22944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:31.839600 22944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:31.839805 22944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:31.844660 22959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:31.844743 22944 server_base.cc:1034] running on GCE node
W20260810 21:12:31.844702 22954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:31.844660 22961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:31.845115 22944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:31.845366 22944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:31.846532 22944 hybrid_clock.cc:648] HybridClock initialized: now 1786396351846528 us; error 35 us; skew 500 ppm
Aug 10 21:12:31 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:31.849069 22944 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:31.849439 22944 webserver.cc:460] Webserver started at http://127.27.117.65:41611/ using document root <none> and password file <none>
I20260810 21:12:31.849632 22944 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:31.849673 22944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:31.849817 22944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:31.850803 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "743ecfdb611c49aab54dff0e7018bccb"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "a632e0995c99986de9977d2dbf686b43"
server_key_iv: "79037ef7c7963ebaf12775c0854ea2af"
server_key_version: "encryptionkey@0"
I20260810 21:12:31.851116 22944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "743ecfdb611c49aab54dff0e7018bccb"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "a632e0995c99986de9977d2dbf686b43"
server_key_iv: "79037ef7c7963ebaf12775c0854ea2af"
server_key_version: "encryptionkey@0"
I20260810 21:12:31.852357 22944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:12:31.853082 22969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:31.853240 22944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:31.853327 22944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "743ecfdb611c49aab54dff0e7018bccb"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "a632e0995c99986de9977d2dbf686b43"
server_key_iv: "79037ef7c7963ebaf12775c0854ea2af"
server_key_version: "encryptionkey@0"
I20260810 21:12:31.853399 22944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:31.864468 22944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:31.866806 22944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:31.866935 22944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:31.867159 22944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:31.867483 22944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:31.867538 22944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:31.867580 22944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:31.867604 22944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:31.873591 22944 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37733
I20260810 21:12:31.873703 23161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37733 every 8 connection(s)
I20260810 21:12:31.873992 22944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:12:31 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:31.880373 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 22944
I20260810 21:12:31.880460 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:12:31.880690 28117 external_mini_cluster.cc:1428] Setting key 8c18cab376b3b247c3bd570795424169
I20260810 21:12:31.883828 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:31.874744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:59913 (local address 127.27.117.126:45155)
0810 21:12:31.874809 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:31.874811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:31.875008 (+ 197us) server_negotiation.cc:407] Connection header received
0810 21:12:31.875560 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:31.875561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:31.875575 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:31.875637 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:31.876596 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:31.877210 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:31.878212 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:31.878433 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:31.878512 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:31.880862 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:31.880864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:31.880864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:31.880874 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:31.882322 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:31.882747 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:31.882748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:31.882749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:31.882816 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:31.883123 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:31.883125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:31.883125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:31.883284 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:31.883347 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:31.883687 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 21:12:31.883763 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:12:31.884166 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:31.874721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:59913)
0810 21:12:31.874884 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:31.874921 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:31.875441 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:31.875713 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:31.875716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:31.876300 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:31.876488 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:31.876492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:31.877307 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:31.877308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:31.878114 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:31.878116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:31.878522 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:31.878523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:31.878652 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:31.879130 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:31.879139 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:31.880702 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:31.882431 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:31.882439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:31.882441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:31.882665 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:31.882886 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:31.882887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:31.882887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:31.883036 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:31.883359 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:31.883366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:31.883546 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 21:12:31.883673 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 21:12:31.883856 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260810 21:12:31.884739 23163 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45155
I20260810 21:12:31.884886 23163 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:31.885071 23163 heartbeater.cc:511] Master 127.27.117.126:45155 requested a full tablet report, sending...
I20260810 21:12:31.885635 22817 ts_manager.cc:194] Registered new tserver with Master: 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733)
I20260810 21:12:31.886411 22817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:59913
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:12:31.905869 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45155
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:31.988359 23168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:31.988574 23168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:31.988600 23168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:31.989813 23168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:31.989862 23168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:31.989889 23168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:31.991415 23168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:31.991667 23168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:31.991906 23168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:31.996984 23176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:31.997030 23174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:31.997030 23173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:31.997140 23168 server_base.cc:1034] running on GCE node
I20260810 21:12:31.997381 23168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:31.997640 23168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:31.998793 23168 hybrid_clock.cc:648] HybridClock initialized: now 1786396351998776 us; error 33 us; skew 500 ppm
Aug 10 21:12:31 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:32.001472 23168 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:32.001986 23168 webserver.cc:460] Webserver started at http://127.27.117.66:36261/ using document root <none> and password file <none>
I20260810 21:12:32.002288 23168 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:32.002336 23168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:32.002485 23168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:32.003614 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "4a531952097d4f2d9b8f2dc4efd55eb5"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "716750832bbee9539879a3d76f01873e"
server_key_iv: "c976758ea1246f738f158cdb82a1cbb4"
server_key_version: "encryptionkey@0"
I20260810 21:12:32.004009 23168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "4a531952097d4f2d9b8f2dc4efd55eb5"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "716750832bbee9539879a3d76f01873e"
server_key_iv: "c976758ea1246f738f158cdb82a1cbb4"
server_key_version: "encryptionkey@0"
I20260810 21:12:32.005566 23168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:12:32.006295 23184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:32.006474 23168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:32.006570 23168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "4a531952097d4f2d9b8f2dc4efd55eb5"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "716750832bbee9539879a3d76f01873e"
server_key_iv: "c976758ea1246f738f158cdb82a1cbb4"
server_key_version: "encryptionkey@0"
I20260810 21:12:32.006656 23168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:32.019552 23168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:32.021982 23168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:32.022167 23168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:32.022426 23168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:32.022750 23168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:32.022807 23168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:32.022858 23168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:32.022881 23168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:32.028959 23168 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:45281
I20260810 21:12:32.029011 23380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:45281 every 8 connection(s)
I20260810 21:12:32.029393 23168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:12:32.030944 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 23168
I20260810 21:12:32.031045 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:12:32.031216 28117 external_mini_cluster.cc:1428] Setting key 5b4d7aa90194c379b25389fd452bad14
Aug 10 21:12:32 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:32.039098 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.030104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:48003 (local address 127.27.117.126:45155)
0810 21:12:32.030171 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.030173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.030404 (+ 231us) server_negotiation.cc:407] Connection header received
0810 21:12:32.030962 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.030964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.030976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.031032 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:32.032193 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.032690 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.033728 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.033921 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.034009 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.036208 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:32.036210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:32.036210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:32.036219 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:32.037695 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:32.038098 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:32.038099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:32.038100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:32.038162 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:32.038429 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:32.038430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:32.038430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:32.038544 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:32.038626 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.038910 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.038997 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:12:32.039264 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.030066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:48003)
0810 21:12:32.030259 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.030302 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.030849 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.031107 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.031111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.031816 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:32.032099 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.032105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.032809 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.032810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.033633 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.033636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.034023 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.034024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.034161 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.034702 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:32.034710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:32.036105 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:32.037780 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:32.037787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:32.037789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:32.038012 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:32.038238 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:32.038239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:32.038239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:32.038351 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:32.038615 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:32.038622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:32.038764 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.038870 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.039029 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
I20260810 21:12:32.039739 23381 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45155
I20260810 21:12:32.039831 23381 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:32.040014 23381 heartbeater.cc:511] Master 127.27.117.126:45155 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
I20260810 21:12:32.040447 22817 ts_manager.cc:194] Registered new tserver with Master: 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
I20260810 21:12:32.040946 22817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:48003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:12:32.057699 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45155
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:32.139412 23390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:32.139622 23390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:32.139647 23390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:32.140825 23390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:32.140865 23390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:32.140892 23390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:32.142359 23390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:32.142613 23390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:32.142840 23390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:12:32.148005 23390 server_base.cc:1034] running on GCE node
W20260810 21:12:32.147979 23404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:32.148061 23398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:32.147980 23400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:32.148391 23390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:32.148662 23390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:32.149806 23390 hybrid_clock.cc:648] HybridClock initialized: now 1786396352149793 us; error 30 us; skew 500 ppm
Aug 10 21:12:32 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:32.152199 23390 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:32.152642 23390 webserver.cc:460] Webserver started at http://127.27.117.67:40961/ using document root <none> and password file <none>
I20260810 21:12:32.152940 23390 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:32.153012 23390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:32.153177 23390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:12:32.154179 23390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "b0f1d25602354afd91ed47aa3ea506bd"
format_stamp: "Formatted at 2026-08-10 21:12:32 on dist-test-slave-2kcd"
server_key: "f692a993d63a74720bf625038f7aea9d"
server_key_iv: "bdaeb7973e6cd1f496e210b8a9cfc10f"
server_key_version: "encryptionkey@0"
I20260810 21:12:32.154543 23390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "b0f1d25602354afd91ed47aa3ea506bd"
format_stamp: "Formatted at 2026-08-10 21:12:32 on dist-test-slave-2kcd"
server_key: "f692a993d63a74720bf625038f7aea9d"
server_key_iv: "bdaeb7973e6cd1f496e210b8a9cfc10f"
server_key_version: "encryptionkey@0"
I20260810 21:12:32.155963 23390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:12:32.156759 23419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:32.156993 23390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:32.157104 23390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "b0f1d25602354afd91ed47aa3ea506bd"
format_stamp: "Formatted at 2026-08-10 21:12:32 on dist-test-slave-2kcd"
server_key: "f692a993d63a74720bf625038f7aea9d"
server_key_iv: "bdaeb7973e6cd1f496e210b8a9cfc10f"
server_key_version: "encryptionkey@0"
I20260810 21:12:32.157186 23390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:32.184728 23390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:32.186679 23390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:32.186854 23390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:32.187119 23390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:32.187458 23390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:32.187513 23390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:32.187551 23390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:32.187598 23390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:32.193758 23390 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:42855
I20260810 21:12:32.193805 23634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:42855 every 8 connection(s)
I20260810 21:12:32.194128 23390 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:12:32 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:32.202895 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.194850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:47523 (local address 127.27.117.126:45155)
0810 21:12:32.194943 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.194946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.195122 (+ 176us) server_negotiation.cc:407] Connection header received
0810 21:12:32.195586 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.195587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.195597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.195650 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:32.196553 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.197116 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.198064 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.198255 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.198312 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.200392 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:32.200394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:32.200394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:32.200400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:32.201661 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:32.202010 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:32.202010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:32.202011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:32.202060 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:32.202304 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:32.202305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:32.202306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:32.202458 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:32.202523 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.202765 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.202828 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 21:12:32.203109 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.194808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:47523)
0810 21:12:32.194996 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.195041 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.195507 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.195702 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.195705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.196282 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:32.196476 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.196480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.197194 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.197195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.197974 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.197976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.198330 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.198331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.198434 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.198958 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:32.198967 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:32.200307 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:32.201730 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:32.201736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:32.201738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:32.201946 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:32.202116 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:32.202117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:32.202117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:32.202220 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:32.202529 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:32.202534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:32.202650 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.202737 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.202854 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260810 21:12:32.203393 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 23390
I20260810 21:12:32.203521 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:12:32.203717 23635 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45155
I20260810 21:12:32.203753 28117 external_mini_cluster.cc:1428] Setting key dcb883b9fc105e5821dc0f29a550c0b7
I20260810 21:12:32.203836 23635 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:32.204043 23635 heartbeater.cc:511] Master 127.27.117.126:45155 requested a full tablet report, sending...
I20260810 21:12:32.204427 22817 ts_manager.cc:194] Registered new tserver with Master: b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855)
I20260810 21:12:32.204828 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:12:32.205056 22817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:47523
I20260810 21:12:32.212038 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.206385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.27.117.126:45155)
0810 21:12:32.206457 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.206458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.206574 (+ 116us) server_negotiation.cc:407] Connection header received
0810 21:12:32.206619 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.206619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.206627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.206680 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:32.207181 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.207623 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.208306 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.208474 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.208529 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.209714 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:32.209716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:32.209716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:32.209723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:32.211107 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:32.211410 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:32.211411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:32.211411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:32.211460 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:32.211624 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:32.211624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:32.211625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:32.211743 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:32.211819 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.211925 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.211980 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:12:32.213685 22817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:12:32.215173 22817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:12:32.221352 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.217320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42855 (local address 127.0.0.1:59038)
0810 21:12:32.217419 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.217455 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.217538 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.217973 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.217976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.217979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:32.218046 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.218048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.219074 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.219075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.220185 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.220188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.221033 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.221034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.221125 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.221138 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.221234 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.221281 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":56,"thread_start_us":56,"threads_started":1}
I20260810 21:12:32.221395 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.217586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.27.117.67:42855)
0810 21:12:32.217775 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.217780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.217792 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:12:32.217831 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.217833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.217850 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.217940 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:32.218193 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.218959 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.220276 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.220948 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.221022 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.221078 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.221264 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.221315 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260810 21:12:32.221688 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.217342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:45281 (local address 127.0.0.1:49668)
0810 21:12:32.217792 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.217801 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.217841 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.218078 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.218079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.218082 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:32.218132 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.218134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.219066 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.219067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.220420 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.220423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.221435 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.221437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.221506 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.221510 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.221575 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.221615 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":33,"thread_start_us":54,"threads_started":1}
I20260810 21:12:32.221717 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.217599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.27.117.66:45281)
0810 21:12:32.217852 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.217855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.217865 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:12:32.217906 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.217908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.217923 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.217994 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:32.218213 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.218951 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.220513 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.221280 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.221346 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.221397 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.221594 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.221636 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
I20260810 21:12:32.221999 23507 tablet_service.cc:1467] Processing CreateTablet for tablet 67fbd929c22448359f71281ed15932a8 (DEFAULT_TABLE table=test-table [id=6b34e6d434d84782b5a2eb1fa0a95346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:32.222317 23507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fbd929c22448359f71281ed15932a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:32.223125 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.217342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37733 (local address 127.0.0.1:53132)
0810 21:12:32.217764 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.217775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.217831 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.219311 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.219312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.219314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:32.219380 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.219381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.220558 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.220559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.221988 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.221991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.222880 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.222881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.222942 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.222946 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.223024 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.223061 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":65,"threads_started":1}
I20260810 21:12:32.223145 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.217601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.27.117.65:37733)
0810 21:12:32.219153 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.219158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.219167 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:32.219203 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.219206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.219221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.219300 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:32.219442 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.220471 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.222087 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.222782 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.222879 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.222931 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.223037 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.223071 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516,"thread_start_us":114,"threads_started":1}
I20260810 21:12:32.223990 23051 tablet_service.cc:1467] Processing CreateTablet for tablet 67fbd929c22448359f71281ed15932a8 (DEFAULT_TABLE table=test-table [id=6b34e6d434d84782b5a2eb1fa0a95346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:32.224295 23051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fbd929c22448359f71281ed15932a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:32.225538 23663 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Bootstrap starting.
I20260810 21:12:32.225536 23275 tablet_service.cc:1467] Processing CreateTablet for tablet 67fbd929c22448359f71281ed15932a8 (DEFAULT_TABLE table=test-table [id=6b34e6d434d84782b5a2eb1fa0a95346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:32.225889 23275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fbd929c22448359f71281ed15932a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:32.226203 23663 tablet_bootstrap.cc:654] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:32.226709 23663 log.cc:824] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:32.227474 23672 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap starting.
I20260810 21:12:32.227640 23663 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: No bootstrap required, opened a new log
I20260810 21:12:32.227711 23663 ts_tablet_manager.cc:1397] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:32.228168 23672 tablet_bootstrap.cc:654] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:32.228853 23672 log.cc:824] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:32.229409 23673 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Bootstrap starting.
I20260810 21:12:32.229585 23672 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: No bootstrap required, opened a new log
I20260810 21:12:32.229676 23672 ts_tablet_manager.cc:1397] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 21:12:32.230072 23673 tablet_bootstrap.cc:654] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:32.230243 23663 raft_consensus.cc:348] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.230453 23663 raft_consensus.cc:374] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:32.230504 23663 raft_consensus.cc:729] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f1d25602354afd91ed47aa3ea506bd, State: Initialized, Role: FOLLOWER
I20260810 21:12:32.230628 23673 log.cc:824] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:32.230620 23663 consensus_queue.cc:260] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.230890 23663 ts_tablet_manager.cc:1428] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:12:32.231040 23635 heartbeater.cc:503] Master 127.27.117.126:45155 was elected leader, sending a full tablet report...
I20260810 21:12:32.231400 23673 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: No bootstrap required, opened a new log
I20260810 21:12:32.231472 23673 ts_tablet_manager.cc:1397] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:32.231510 23672 raft_consensus.cc:348] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.231678 23672 raft_consensus.cc:374] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:32.231724 23672 raft_consensus.cc:729] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743ecfdb611c49aab54dff0e7018bccb, State: Initialized, Role: FOLLOWER
I20260810 21:12:32.231853 23672 consensus_queue.cc:260] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.232123 23672 ts_tablet_manager.cc:1428] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 21:12:32.232134 23163 heartbeater.cc:503] Master 127.27.117.126:45155 was elected leader, sending a full tablet report...
I20260810 21:12:32.233436 23673 raft_consensus.cc:348] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.233592 23673 raft_consensus.cc:374] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:32.233639 23673 raft_consensus.cc:729] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a531952097d4f2d9b8f2dc4efd55eb5, State: Initialized, Role: FOLLOWER
I20260810 21:12:32.233752 23673 consensus_queue.cc:260] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.234001 23673 ts_tablet_manager.cc:1428] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 21:12:32.234089 23381 heartbeater.cc:503] Master 127.27.117.126:45155 was elected leader, sending a full tablet report...
W20260810 21:12:32.279989 23382 tablet.cc:2369] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:12:32.316392 23685 raft_consensus.cc:482] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:32.316550 23685 raft_consensus.cc:504] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.316972 23685 leader_election.cc:288] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733)
I20260810 21:12:32.321012 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.317239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:44877 (local address 127.27.117.65:37733)
0810 21:12:32.317285 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.317287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.317419 (+ 132us) server_negotiation.cc:407] Connection header received
0810 21:12:32.317448 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.317450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.317459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.317516 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:32.317679 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.318452 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.319989 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.320589 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.320668 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.320677 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.320881 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.320931 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:12:32.321034 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.317147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42855 (local address 127.27.117.66:42733)
0810 21:12:32.317206 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.317221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.317273 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.317485 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.317485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.317487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:32.317590 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.317592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.318534 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.318534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.319882 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.319886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.320674 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.320675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.320766 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.320769 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.320862 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.320924 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":116}
I20260810 21:12:32.321089 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.317287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:42733 (local address 127.27.117.67:42855)
0810 21:12:32.317336 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.317337 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.317353 (+ 16us) server_negotiation.cc:407] Connection header received
0810 21:12:32.317377 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.317377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.317384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.317468 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:32.317682 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.318443 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.319989 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.320589 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.320668 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.320677 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.320916 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.320994 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:12:32.321040 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.317159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37733 (local address 127.27.117.66:44877)
0810 21:12:32.317341 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.317349 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:32.317402 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:32.317534 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:32.317535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:32.317536 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:32.317596 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.317597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.318534 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.318534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.319883 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:32.319886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.320675 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.320675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:32.320756 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.320759 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:32.320862 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 21:12:32.320924 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":71,"thread_start_us":118,"threads_started":1}
I20260810 21:12:32.321475 23540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fbd929c22448359f71281ed15932a8" candidate_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f1d25602354afd91ed47aa3ea506bd" is_pre_election: true
I20260810 21:12:32.321631 23540 raft_consensus.cc:2454] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a531952097d4f2d9b8f2dc4efd55eb5 in term 0.
I20260810 21:12:32.321659 23082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fbd929c22448359f71281ed15932a8" candidate_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "743ecfdb611c49aab54dff0e7018bccb" is_pre_election: true
I20260810 21:12:32.321815 23082 raft_consensus.cc:2454] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a531952097d4f2d9b8f2dc4efd55eb5 in term 0.
I20260810 21:12:32.321874 23185 leader_election.cc:302] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a531952097d4f2d9b8f2dc4efd55eb5, b0f1d25602354afd91ed47aa3ea506bd; no voters:
I20260810 21:12:32.322120 23685 raft_consensus.cc:2781] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:12:32.322190 23685 raft_consensus.cc:482] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:12:32.322227 23685 raft_consensus.cc:3037] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:32.322950 23685 raft_consensus.cc:504] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.323112 23685 leader_election.cc:288] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [CANDIDATE]: Term 1 election: Requested vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733)
I20260810 21:12:32.323302 23540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fbd929c22448359f71281ed15932a8" candidate_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f1d25602354afd91ed47aa3ea506bd"
I20260810 21:12:32.323336 23082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fbd929c22448359f71281ed15932a8" candidate_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "743ecfdb611c49aab54dff0e7018bccb"
I20260810 21:12:32.323407 23540 raft_consensus.cc:3037] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:32.323434 23082 raft_consensus.cc:3037] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:32.324204 23540 raft_consensus.cc:2454] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a531952097d4f2d9b8f2dc4efd55eb5 in term 1.
I20260810 21:12:32.324205 23082 raft_consensus.cc:2454] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a531952097d4f2d9b8f2dc4efd55eb5 in term 1.
I20260810 21:12:32.324427 23189 leader_election.cc:302] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a531952097d4f2d9b8f2dc4efd55eb5, 743ecfdb611c49aab54dff0e7018bccb; no voters:
I20260810 21:12:32.324525 23685 raft_consensus.cc:2781] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:32.324723 23685 raft_consensus.cc:686] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 4a531952097d4f2d9b8f2dc4efd55eb5, State: Running, Role: LEADER
I20260810 21:12:32.324882 23685 consensus_queue.cc:237] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:32.325809 22814 catalog_manager.cc:5495] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66). New cstate: current_term: 1 leader_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } health_report { overall_health: UNKNOWN } } }
I20260810 21:12:32.335608 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.333511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.27.117.66:45281)
0810 21:12:32.333571 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:32.333575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:32.333588 (+ 13us) server_negotiation.cc:407] Connection header received
0810 21:12:32.333617 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:32.333618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:32.333633 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:32.333693 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:32.333817 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.334268 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.334949 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:32.335129 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:32.335177 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:32.335307 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:32.335394 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:32.335451 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:32.335482 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 21:12:32.335499 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 21:12:32.337330 23540 raft_consensus.cc:1261] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Refusing update from remote peer 4a531952097d4f2d9b8f2dc4efd55eb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:32.337348 23082 raft_consensus.cc:1261] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Refusing update from remote peer 4a531952097d4f2d9b8f2dc4efd55eb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:32.337682 23685 consensus_queue.cc:1035] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:32.337846 23690 consensus_queue.cc:1035] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:32.340116 23696 mvcc.cc:205] Tried to move back new op lower bound from 7317079459170959360 to 7317079459123380224. Current Snapshot: MvccSnapshot[applied={T|T < 7317079459170959360}]
I20260810 21:12:32.340515 23702 mvcc.cc:205] Tried to move back new op lower bound from 7317079459170959360 to 7317079459123380224. Current Snapshot: MvccSnapshot[applied={T|T < 7317079459170959360}]
I20260810 21:12:32.340546 23703 mvcc.cc:205] Tried to move back new op lower bound from 7317079459170959360 to 7317079459123380224. Current Snapshot: MvccSnapshot[applied={T|T < 7317079459170959360}]
I20260810 21:12:32.343185 22814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:32.343274 22814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:32.344081 22814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Sending DeleteTablet for 3 replicas of tablet 67fbd929c22448359f71281ed15932a8
I20260810 21:12:32.344448 23507 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fbd929c22448359f71281ed15932a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:32 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:59038
I20260810 21:12:32.344485 23275 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fbd929c22448359f71281ed15932a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:32 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:49668
I20260810 21:12:32.344727 23716 tablet_replica.cc:330] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: stopping tablet replica
I20260810 21:12:32.344755 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 22944
I20260810 21:12:32.344766 23051 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fbd929c22448359f71281ed15932a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:32 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:53132
I20260810 21:12:32.355360 23716 raft_consensus.cc:2229] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:12:32.355638 23716 raft_consensus.cc:2258] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:32.356894 23715 tablet_replica.cc:330] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: stopping tablet replica
I20260810 21:12:32.357039 23715 raft_consensus.cc:2229] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:32.357160 23715 raft_consensus.cc:2258] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:32.357245 23716 ts_tablet_manager.cc:1905] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:32.357642 23715 ts_tablet_manager.cc:1905] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:32.359386 23715 ts_tablet_manager.cc:1918] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:32.359475 23715 log.cc:1196] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/67fbd929c22448359f71281ed15932a8
I20260810 21:12:32.359773 23716 ts_tablet_manager.cc:1918] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:32.359881 23715 ts_tablet_manager.cc:1939] T 67fbd929c22448359f71281ed15932a8 P b0f1d25602354afd91ed47aa3ea506bd: Deleting consensus metadata
I20260810 21:12:32.359968 23716 log.cc:1196] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/67fbd929c22448359f71281ed15932a8
I20260810 21:12:32.360317 23716 ts_tablet_manager.cc:1939] T 67fbd929c22448359f71281ed15932a8 P 4a531952097d4f2d9b8f2dc4efd55eb5: Deleting consensus metadata
I20260810 21:12:32.360416 22790 catalog_manager.cc:4841] TS b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): tablet 67fbd929c22448359f71281ed15932a8 (table test-table [id=6b34e6d434d84782b5a2eb1fa0a95346]) successfully deleted
I20260810 21:12:32.360829 22794 catalog_manager.cc:4841] TS 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): tablet 67fbd929c22448359f71281ed15932a8 (table test-table [id=6b34e6d434d84782b5a2eb1fa0a95346]) successfully deleted
W20260810 21:12:32.363517 22796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:37733 (error 108)
I20260810 21:12:32.364094 23654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:32.363933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37733 (local address 127.0.0.1:53144)
0810 21:12:32.363999 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:32.364048 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 21:12:32.364169 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 23168
W20260810 21:12:32.375536 22796 catalog_manager.cc:4584] TS 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67fbd929c22448359f71281ed15932a8: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37733: connect: Connection refused (error 111)
I20260810 21:12:32.380920 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 23390
I20260810 21:12:32.397344 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 22763
I20260810 21:12:52.413494 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45155
--webserver_interface=127.27.117.126
--webserver_port=34061
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:12:52.499753 23719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:52.500022 23719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:52.500063 23719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:52.501420 23719 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 21:12:52.501495 23719 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 21:12:52.501520 23719 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 21:12:52.501538 23719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:12:52.501571 23719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:52.501596 23719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:52.501621 23719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:12:52.501644 23719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:12:52.503270 23719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:52.503611 23719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:52.503921 23719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:52.514176 23728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.514163 23730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.514158 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:52.514235 23719 server_base.cc:1034] running on GCE node
I20260810 21:12:52.514744 23719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:52.515070 23719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:52.516235 23719 hybrid_clock.cc:648] HybridClock initialized: now 1786396372516225 us; error 30 us; skew 500 ppm
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:52.521727 23719 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:12:52.522241 23719 webserver.cc:460] Webserver started at http://127.27.117.126:34061/ using document root <none> and password file <none>
I20260810 21:12:52.522507 23719 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:52.522599 23719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:52.523988 23719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:12:52.524689 23744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.524927 23719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:52.525050 23719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "c1645a25315c47f1b501a78fe2a2ec6f"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "6d0e1713615dcec612b57cdf97e6871b"
server_key_iv: "4ee9f57855b1b204825a024ee274ef5a"
server_key_version: "encryptionkey@0"
I20260810 21:12:52.525447 23719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:52.546679 23719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:52.548538 23719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:52.548718 23719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:52.553676 23719 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:45155
I20260810 21:12:52.553709 23843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:45155 every 8 connection(s)
I20260810 21:12:52.554111 23719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:12:52.555722 23844 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:12:52.556386 23844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap starting.
I20260810 21:12:52.559705 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 23719
I20260810 21:12:52.560053 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:37733
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:45155
--webserver_port=41611
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:12:52.566191 23844 log.cc:824] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:52.568140 23844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:52.568454 23844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap complete.
I20260810 21:12:52.570827 23844 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:52.571045 23844 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1645a25315c47f1b501a78fe2a2ec6f, State: Initialized, Role: FOLLOWER
I20260810 21:12:52.571214 23844 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:52.571274 23844 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:12:52.571326 23844 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:12:52.571376 23844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 1 FOLLOWER]: Advancing to term 2
I20260810 21:12:52.572356 23844 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:52.572479 23844 leader_election.cc:302] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1645a25315c47f1b501a78fe2a2ec6f; no voters:
I20260810 21:12:52.572681 23844 leader_election.cc:288] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 21:12:52.572731 23852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Leader election won for term 2
I20260810 21:12:52.572883 23852 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 LEADER]: Becoming Leader. State: Replica: c1645a25315c47f1b501a78fe2a2ec6f, State: Running, Role: LEADER
I20260810 21:12:52.572983 23844 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:12:52.572985 23852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:12:52.573385 23852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1645a25315c47f1b501a78fe2a2ec6f. Latest consensus state: current_term: 2 leader_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } } }
I20260810 21:12:52.573478 23852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:52.574164 23853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } } }
I20260810 21:12:52.574255 23853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: This master's current role is: LEADER
I20260810 21:12:52.574471 23865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:12:52.575388 23865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:12:52.575506 23865 catalog_manager.cc:1259] Loaded cluster ID: 22555892b2dc4d9f87f29f4690311fa0
I20260810 21:12:52.575541 23865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:12:52.576653 23865 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:12:52.576856 23865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Loaded TSK: 0
I20260810 21:12:52.586508 23865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Generated new TSK 1
I20260810 21:12:52.586596 23865 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 21:12:52.646739 23849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:52.646999 23849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:52.647043 23849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:52.648213 23849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:52.648286 23849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:52.648332 23849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:12:52.649926 23849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:37733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=41611
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:52.650218 23849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:52.650470 23849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:52.655772 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:52.655941 23849 server_base.cc:1034] running on GCE node
W20260810 21:12:52.655781 23900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.656042 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:52.656262 23849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:52.656528 23849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:52.657697 23849 hybrid_clock.cc:648] HybridClock initialized: now 1786396372657674 us; error 42 us; skew 500 ppm
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:52.660324 23849 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:12:52.660871 23849 webserver.cc:460] Webserver started at http://127.27.117.65:41611/ using document root <none> and password file <none>
I20260810 21:12:52.661144 23849 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:52.661207 23849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:52.662672 23849 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:12:52.663455 23910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.663619 23849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:52.663740 23849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "743ecfdb611c49aab54dff0e7018bccb"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "a632e0995c99986de9977d2dbf686b43"
server_key_iv: "79037ef7c7963ebaf12775c0854ea2af"
server_key_version: "encryptionkey@0"
I20260810 21:12:52.664091 23849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:52.682729 23849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:52.685369 23849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:52.685547 23849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:52.685848 23849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:52.686475 23923 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 21:12:52.688093 23849 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 21:12:52.688153 23849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 21:12:52.688192 23849 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 21:12:52.689024 23849 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 21:12:52.689076 23849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:52.689121 23923 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap starting.
I20260810 21:12:52.698060 23849 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37733
I20260810 21:12:52.698202 24107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37733 every 8 connection(s)
I20260810 21:12:52.698633 23849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:12:52.701435 23923 log.cc:824] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:52.704013 23923 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:12:52.704361 23923 tablet_bootstrap.cc:492] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap complete.
I20260810 21:12:52.705524 23923 ts_tablet_manager.cc:1397] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260810 21:12:52.706333 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 23849
I20260810 21:12:52.706614 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:45281
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:45155
--webserver_port=36261
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:52.706941 23923 raft_consensus.cc:348] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.707327 23923 raft_consensus.cc:729] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743ecfdb611c49aab54dff0e7018bccb, State: Initialized, Role: FOLLOWER
I20260810 21:12:52.707479 23923 consensus_queue.cc:260] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.707755 23923 ts_tablet_manager.cc:1428] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 21:12:52.712121 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.699379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:44749)
0810 21:12:52.699655 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:52.699706 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:52.700358 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:52.700990 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:52.700995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:52.701332 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:52.701581 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.701587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.703151 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.703152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.704063 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.704065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.704762 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.704764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.704939 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.705512 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:52.705520 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:52.707414 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:52.710414 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.710421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.710423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.710668 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.710938 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.710939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.710939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.711081 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.711483 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:52.711490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:52.711623 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 21:12:52.711732 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 21:12:52.711927 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
I20260810 21:12:52.712121 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.699507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:44749 (local address 127.27.117.126:45155)
0810 21:12:52.699760 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.699770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.699826 (+ 56us) server_negotiation.cc:407] Connection header received
0810 21:12:52.700636 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.700647 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.700718 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.700968 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:52.701698 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.702990 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.704178 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.704621 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.704748 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.708531 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:52.708534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:52.708534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:52.708545 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:52.710304 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.710756 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.710759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.710759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.710859 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.711167 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.711168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.711168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.711401 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:52.711491 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.711799 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.711927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
I20260810 21:12:52.712833 24108 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45155
I20260810 21:12:52.712931 24108 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:52.713121 24108 heartbeater.cc:511] Master 127.27.117.126:45155 requested a full tablet report, sending...
I20260810 21:12:52.713907 23773 ts_manager.cc:194] Registered new tserver with Master: 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733)
I20260810 21:12:52.715251 23773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:44749
I20260810 21:12:52.716168 24108 heartbeater.cc:503] Master 127.27.117.126:45155 was elected leader, sending a full tablet report...
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.65@KRBTEST.COM
I20260810 21:12:52.723531 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.714694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.27.117.65:37733)
0810 21:12:52.715092 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.715096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.715103 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:12:52.715128 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.715130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.715140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.715223 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:52.715790 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.716970 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.717715 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.718213 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.718320 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.720580 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:52.720587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:52.720588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:52.720594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:52.722218 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.722698 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.722700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.722701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.722756 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.722977 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.722978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.722978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.723120 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:52.723183 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.723386 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.723463 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":70,"threads_started":1}
I20260810 21:12:52.723599 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.714642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37733 (local address 127.0.0.1:43762)
0810 21:12:52.714893 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:52.714917 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:52.714974 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:52.715227 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:52.715234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:52.715621 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:52.715688 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.715691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.717059 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.717060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.717624 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.717625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.718305 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.718306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.718361 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.718872 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:52.718878 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:52.720480 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:52.722313 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.722321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.722323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.722610 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.722826 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.722827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.722827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.722903 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.723187 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:52.723194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:52.723250 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 21:12:52.723353 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 21:12:52.723485 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260810 21:12:52.724073 23996 tablet_service.cc:1514] Processing DeleteTablet for tablet 67fbd929c22448359f71281ed15932a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:32 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:43762
I20260810 21:12:52.724606 24121 tablet_replica.cc:330] stopping tablet replica
I20260810 21:12:52.724727 24121 raft_consensus.cc:2229] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:52.724820 24121 raft_consensus.cc:2258] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:52.725148 24121 ts_tablet_manager.cc:1905] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:52.727267 24121 ts_tablet_manager.cc:1918] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:52.727353 24121 log.cc:1196] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal/wals/67fbd929c22448359f71281ed15932a8
I20260810 21:12:52.727626 24121 ts_tablet_manager.cc:1939] T 67fbd929c22448359f71281ed15932a8 P 743ecfdb611c49aab54dff0e7018bccb: Deleting consensus metadata
I20260810 21:12:52.728283 23755 catalog_manager.cc:4841] TS 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733): tablet 67fbd929c22448359f71281ed15932a8 (table test-table [id=6b34e6d434d84782b5a2eb1fa0a95346]) successfully deleted
W20260810 21:12:52.796183 24117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:52.796403 24117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:52.796432 24117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:52.797557 24117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:52.797600 24117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:52.797627 24117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:12:52.799114 24117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:45281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=36261
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:52.799317 24117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:52.799501 24117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:52.804607 24133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.804740 24131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.804617 24137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:52.804891 24117 server_base.cc:1034] running on GCE node
I20260810 21:12:52.805145 24117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:52.805361 24117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:52.806507 24117 hybrid_clock.cc:648] HybridClock initialized: now 1786396372806488 us; error 29 us; skew 500 ppm
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:52.808935 24117 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:12:52.809373 24117 webserver.cc:460] Webserver started at http://127.27.117.66:36261/ using document root <none> and password file <none>
I20260810 21:12:52.809631 24117 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:52.809725 24117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:52.811079 24117 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:52.811664 24145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.811846 24117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:52.811955 24117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "4a531952097d4f2d9b8f2dc4efd55eb5"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "716750832bbee9539879a3d76f01873e"
server_key_iv: "c976758ea1246f738f158cdb82a1cbb4"
server_key_version: "encryptionkey@0"
I20260810 21:12:52.812304 24117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:52.828819 24117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:52.830610 24117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:52.830772 24117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:52.830992 24117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:52.831293 24117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:52.831347 24117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.831386 24117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:52.831414 24117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.837533 24117 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:45281
I20260810 21:12:52.837595 24331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:45281 every 8 connection(s)
I20260810 21:12:52.837955 24117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:12:52.843030 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 24117
I20260810 21:12:52.843319 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:42855
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:45155
--webserver_port=40961
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:52.847534 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.838764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:55059 (local address 127.27.117.126:45155)
0810 21:12:52.838826 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.838828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.839018 (+ 190us) server_negotiation.cc:407] Connection header received
0810 21:12:52.839523 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.839524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.839539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.839592 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:52.840520 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.841029 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.841975 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.842178 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.842229 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.844606 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:52.844607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:52.844608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:52.844616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:52.846065 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.846528 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.846529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.846529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.846583 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.846850 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.846851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.846851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.846978 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:52.847058 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.847326 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.847442 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 21:12:52.847698 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.838760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:55059)
0810 21:12:52.838911 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:52.838946 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:52.839426 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:52.839668 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:52.839671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:52.840237 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:52.840436 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.840440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.841108 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.841109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.841898 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.841900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.842249 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.842249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.842343 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.842892 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:52.842903 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:52.844477 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:52.846153 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.846162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.846166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.846463 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.846671 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.846671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.846672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.846779 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.847060 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:52.847067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:52.847196 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 21:12:52.847303 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 21:12:52.847465 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260810 21:12:52.848218 24332 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45155
I20260810 21:12:52.848316 24332 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:52.848526 24332 heartbeater.cc:511] Master 127.27.117.126:45155 requested a full tablet report, sending...
I20260810 21:12:52.848978 23773 ts_manager.cc:194] Registered new tserver with Master: 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
I20260810 21:12:52.849359 23773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:55059
W20260810 21:12:52.927151 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:12:52.927373 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:12:52.927412 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:12:52.928607 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:12:52.928676 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:12:52.928721 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:12:52.930279 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:42855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=40961
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:12:52.930545 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:12:52.930768 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:12:52.935750 24345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.935771 24343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:12:52.935761 24342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:12:52.935986 24336 server_base.cc:1034] running on GCE node
I20260810 21:12:52.936211 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:12:52.936507 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:12:52.937690 24336 hybrid_clock.cc:648] HybridClock initialized: now 1786396372937675 us; error 25 us; skew 500 ppm
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:12:52.940181 24336 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:12:52.940619 24336 webserver.cc:460] Webserver started at http://127.27.117.67:40961/ using document root <none> and password file <none>
I20260810 21:12:52.940919 24336 fs_manager.cc:359] Metadata directory not provided
I20260810 21:12:52.941005 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:12:52.942302 24336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:12:52.942898 24357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.943073 24336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:12:52.943174 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "b0f1d25602354afd91ed47aa3ea506bd"
format_stamp: "Formatted at 2026-08-10 21:12:32 on dist-test-slave-2kcd"
server_key: "f692a993d63a74720bf625038f7aea9d"
server_key_iv: "bdaeb7973e6cd1f496e210b8a9cfc10f"
server_key_version: "encryptionkey@0"
I20260810 21:12:52.943501 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:12:52.953292 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:12:52.954996 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:12:52.955153 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:12:52.955365 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:12:52.955674 24336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:12:52.955729 24336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.955770 24336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:12:52.955797 24336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:12:52.961782 24336 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:42855
I20260810 21:12:52.961858 24580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:42855 every 8 connection(s)
I20260810 21:12:52.962198 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
Aug 10 21:12:52 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:12:52.969988 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 24336
I20260810 21:12:52.971699 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.962969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:56161 (local address 127.27.117.126:45155)
0810 21:12:52.963041 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.963043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.963247 (+ 204us) server_negotiation.cc:407] Connection header received
0810 21:12:52.963770 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.963772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.963784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.963834 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:52.964813 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.965319 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.966312 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.966545 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.966636 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.968812 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:52.968813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:52.968814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:52.968821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:52.970111 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.970626 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.970628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.970628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.970701 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.970989 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.970989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.970990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.971149 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:52.971228 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.971485 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.971579 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:12:52.971805 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.962960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:56161)
0810 21:12:52.963128 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:52.963162 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:52.963653 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:52.963904 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:52.963907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:52.964500 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:12:52.964685 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.964690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.965419 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.965420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.966215 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.966217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.966649 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.966650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.966776 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.967335 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:12:52.967343 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:12:52.968685 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:12:52.970212 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.970220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.970223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.970539 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.970768 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:12:52.970769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:12:52.970769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:12:52.970910 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:12:52.971221 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:12:52.971226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:12:52.971338 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 21:12:52.971421 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 21:12:52.971539 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260810 21:12:52.972211 24581 heartbeater.cc:347] Connected to a master server at 127.27.117.126:45155
I20260810 21:12:52.972306 24581 heartbeater.cc:464] Registering TS with master...
I20260810 21:12:52.972508 24581 heartbeater.cc:511] Master 127.27.117.126:45155 requested a full tablet report, sending...
I20260810 21:12:52.972996 23773 ts_manager.cc:194] Registered new tserver with Master: b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855)
I20260810 21:12:52.973479 23773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:56161
I20260810 21:12:52.977236 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.970426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.27.117.126:45155)
0810 21:12:52.970557 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.970559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.970659 (+ 100us) server_negotiation.cc:407] Connection header received
0810 21:12:52.970764 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.970764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.970776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.970842 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:52.971597 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.972351 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.973481 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.973785 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.973836 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.975114 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:52.975115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:52.975115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:52.975121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:52.976269 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.976568 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.976568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.976569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.976619 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:52.976824 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:52.976825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:52.976826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:52.976935 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:52.977060 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.977101 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.977165 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260810 21:12:52.977917 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:12:52.980357 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.978260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.27.117.126:45155)
0810 21:12:52.978295 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.978297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.978424 (+ 127us) server_negotiation.cc:407] Connection header received
0810 21:12:52.978470 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.978471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.978496 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.978554 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:52.978633 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.979051 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.979755 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.979928 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.979975 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.980101 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:52.980178 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:52.980235 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.980270 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.980286 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:12:52.980628 23773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:12:52.982264 23773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:12:52.984640 23996 tablet_service.cc:1467] Processing CreateTablet for tablet fe5bc92c43e94e3daa843d4127e260dc (DEFAULT_TABLE table=test-table [id=fc39c0feeb3147ba99189810db880409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:52.984987 23996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5bc92c43e94e3daa843d4127e260dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:52.986420 23923 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap starting.
I20260810 21:12:52.986781 23923 tablet_bootstrap.cc:654] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:52.987419 23923 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: No bootstrap required, opened a new log
I20260810 21:12:52.987476 23923 ts_tablet_manager.cc:1397] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:52.987823 23923 raft_consensus.cc:348] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.987912 23923 raft_consensus.cc:374] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:52.987974 23923 raft_consensus.cc:729] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743ecfdb611c49aab54dff0e7018bccb, State: Initialized, Role: FOLLOWER
I20260810 21:12:52.988061 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.984053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42855 (local address 127.0.0.1:50022)
0810 21:12:52.984106 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:52.984116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:52.984159 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:52.984601 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:52.984602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:52.984604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:52.984650 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.984652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.985878 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.985879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.986959 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.986962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.987798 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.987800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.987874 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.987894 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 21:12:52.987945 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:12:52.987987 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 21:12:52.988039 23923 consensus_queue.cc:260] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.988152 23923 ts_tablet_manager.cc:1428] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 21:12:52.988233 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.984103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:45281 (local address 127.0.0.1:59542)
0810 21:12:52.984257 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:52.984266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:52.984308 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:52.984701 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:52.984702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:52.984704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:52.984742 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.984743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.985765 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.985766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.986988 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:52.986990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.987997 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.987998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:52.988066 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.988070 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:12:52.988113 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 21:12:52.988154 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":31,"thread_start_us":96,"threads_started":1}
I20260810 21:12:52.988371 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.984116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.27.117.67:42855)
0810 21:12:52.984427 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.984431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.984442 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:12:52.984475 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.984478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.984493 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.984578 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:52.984729 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.985791 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.987063 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.987701 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.987786 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.987842 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.988239 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.988280 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":80,"threads_started":1}
I20260810 21:12:52.988590 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:52.984338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.27.117.66:45281)
0810 21:12:52.984494 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:52.984498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:52.984511 (+ 13us) server_negotiation.cc:407] Connection header received
0810 21:12:52.984572 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:52.984574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:52.984591 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:52.984852 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:52.984941 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.985676 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.987082 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:52.987883 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:52.988338 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:52.988416 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:52.988443 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 21:12:52.988485 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260810 21:12:52.989161 24448 tablet_service.cc:1467] Processing CreateTablet for tablet fe5bc92c43e94e3daa843d4127e260dc (DEFAULT_TABLE table=test-table [id=fc39c0feeb3147ba99189810db880409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:52.989187 24234 tablet_service.cc:1467] Processing CreateTablet for tablet fe5bc92c43e94e3daa843d4127e260dc (DEFAULT_TABLE table=test-table [id=fc39c0feeb3147ba99189810db880409]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:12:52.989477 24448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5bc92c43e94e3daa843d4127e260dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:52.989482 24234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5bc92c43e94e3daa843d4127e260dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:12:52.992419 24602 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Bootstrap starting.
I20260810 21:12:52.992575 24603 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Bootstrap starting.
I20260810 21:12:52.993119 24602 tablet_bootstrap.cc:654] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:52.993271 24603 tablet_bootstrap.cc:654] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Neither blocks nor log segments found. Creating new log.
I20260810 21:12:52.993656 24602 log.cc:824] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:52.993832 24603 log.cc:824] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Log is configured to *not* fsync() on all Append() calls
I20260810 21:12:52.994462 24602 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: No bootstrap required, opened a new log
I20260810 21:12:52.994550 24602 ts_tablet_manager.cc:1397] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:52.994590 24603 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: No bootstrap required, opened a new log
I20260810 21:12:52.994657 24603 ts_tablet_manager.cc:1397] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:52.996100 24603 raft_consensus.cc:348] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.996096 24602 raft_consensus.cc:348] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.996230 24603 raft_consensus.cc:374] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:52.996253 24603 raft_consensus.cc:729] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a531952097d4f2d9b8f2dc4efd55eb5, State: Initialized, Role: FOLLOWER
I20260810 21:12:52.996253 24602 raft_consensus.cc:374] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:12:52.996294 24602 raft_consensus.cc:729] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f1d25602354afd91ed47aa3ea506bd, State: Initialized, Role: FOLLOWER
I20260810 21:12:52.996403 24602 consensus_queue.cc:260] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.996414 24603 consensus_queue.cc:260] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:52.996698 24602 ts_tablet_manager.cc:1428] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:12:52.996721 24332 heartbeater.cc:503] Master 127.27.117.126:45155 was elected leader, sending a full tablet report...
I20260810 21:12:52.996701 24581 heartbeater.cc:503] Master 127.27.117.126:45155 was elected leader, sending a full tablet report...
I20260810 21:12:52.996997 24603 ts_tablet_manager.cc:1428] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260810 21:12:53.088665 24334 tablet.cc:2369] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:12:53.111899 24116 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:12:53.112035 24116 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:53.112426 24116 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
I20260810 21:12:53.115374 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.112720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:57651 (local address 127.27.117.67:42855)
0810 21:12:53.112769 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:53.112771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:53.112808 (+ 37us) server_negotiation.cc:407] Connection header received
0810 21:12:53.112866 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:53.112867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:53.112876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:53.112953 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:53.113127 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.113598 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.114555 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.114991 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.115081 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.115090 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:53.115275 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 21:12:53.115312 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:12:53.115374 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.112616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42855 (local address 127.27.117.65:57651)
0810 21:12:53.112679 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:53.112694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:53.112761 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:53.112961 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:53.112962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:53.112964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:53.113043 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:53.113045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.113683 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.113684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:53.114438 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:53.114440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.115078 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.115079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:53.115163 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.115186 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 21:12:53.115259 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 21:12:53.115302 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 21:12:53.115458 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.112768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:56965 (local address 127.27.117.66:45281)
0810 21:12:53.112850 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:53.112853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:53.112928 (+ 75us) server_negotiation.cc:407] Connection header received
0810 21:12:53.112999 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:53.113000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:53.113009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:53.113073 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:12:53.113242 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.113696 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.114623 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.115064 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.115149 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.115156 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:53.115350 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 21:12:53.115385 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 21:12:53.115505 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.112645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:45281 (local address 127.27.117.65:56965)
0810 21:12:53.112846 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:53.112857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:12:53.112924 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:12:53.113079 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:12:53.113079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:12:53.113081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:12:53.113168 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:53.113170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.113788 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.113789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:53.114532 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:12:53.114533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.115149 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.115150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:12:53.115233 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.115236 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:12:53.115338 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 21:12:53.115379 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":110,"threads_started":1}
I20260810 21:12:53.115743 24485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5bc92c43e94e3daa843d4127e260dc" candidate_uuid: "743ecfdb611c49aab54dff0e7018bccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f1d25602354afd91ed47aa3ea506bd" is_pre_election: true
I20260810 21:12:53.115796 24264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5bc92c43e94e3daa843d4127e260dc" candidate_uuid: "743ecfdb611c49aab54dff0e7018bccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" is_pre_election: true
I20260810 21:12:53.115908 24485 raft_consensus.cc:2454] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 743ecfdb611c49aab54dff0e7018bccb in term 0.
I20260810 21:12:53.115908 24264 raft_consensus.cc:2454] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 743ecfdb611c49aab54dff0e7018bccb in term 0.
I20260810 21:12:53.116158 23912 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb, b0f1d25602354afd91ed47aa3ea506bd; no voters:
I20260810 21:12:53.116317 24116 raft_consensus.cc:2781] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:12:53.116364 24116 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:12:53.116379 24116 raft_consensus.cc:3037] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:53.117136 24116 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:53.117273 24116 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 1 election: Requested vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
I20260810 21:12:53.117432 24485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5bc92c43e94e3daa843d4127e260dc" candidate_uuid: "743ecfdb611c49aab54dff0e7018bccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f1d25602354afd91ed47aa3ea506bd"
I20260810 21:12:53.117450 24264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5bc92c43e94e3daa843d4127e260dc" candidate_uuid: "743ecfdb611c49aab54dff0e7018bccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5"
I20260810 21:12:53.117518 24485 raft_consensus.cc:3037] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:53.117520 24264 raft_consensus.cc:3037] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:12:53.118386 24264 raft_consensus.cc:2454] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 743ecfdb611c49aab54dff0e7018bccb in term 1.
I20260810 21:12:53.118386 24485 raft_consensus.cc:2454] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 743ecfdb611c49aab54dff0e7018bccb in term 1.
I20260810 21:12:53.118552 23912 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb, b0f1d25602354afd91ed47aa3ea506bd; no voters:
I20260810 21:12:53.118654 24116 raft_consensus.cc:2781] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:12:53.118800 24116 raft_consensus.cc:686] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 LEADER]: Becoming Leader. State: Replica: 743ecfdb611c49aab54dff0e7018bccb, State: Running, Role: LEADER
I20260810 21:12:53.118855 24116 consensus_queue.cc:237] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:12:53.119735 23773 catalog_manager.cc:5495] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb reported cstate change: term changed from 0 to 1, leader changed from <none> to 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65). New cstate: current_term: 1 leader_uuid: "743ecfdb611c49aab54dff0e7018bccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } health_report { overall_health: HEALTHY } } }
I20260810 21:12:53.136374 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.134066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.27.117.65:37733)
0810 21:12:53.134119 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:53.134122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:53.134141 (+ 19us) server_negotiation.cc:407] Connection header received
0810 21:12:53.134185 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:53.134186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:53.134203 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:53.134279 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:53.134391 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.134868 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.135631 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.135844 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.135938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.136096 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:53.136148 (+ 52us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 21:12:53.136297 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43776: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 21:12:53.136472 24120 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43776: token expired
I20260810 21:12:53.142155 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.136804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.27.117.126:45155)
0810 21:12:53.136854 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:53.136856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:53.136865 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:12:53.136928 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:53.136928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:53.136936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:53.136983 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:12:53.137423 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.137873 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.138509 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.138682 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.138730 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.139903 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:12:53.139904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:12:53.139904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:12:53.139909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:12:53.141255 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:53.141557 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:53.141558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:53.141558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:53.141611 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:12:53.141775 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:12:53.141775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:12:53.141775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:12:53.141869 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:12:53.141947 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:53.142024 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 21:12:53.142090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 21:12:53.145200 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.143045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.27.117.65:37733)
0810 21:12:53.143083 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 21:12:53.143085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:12:53.143096 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:12:53.143120 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:12:53.143121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:12:53.143131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:12:53.143183 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:12:53.143272 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.143717 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.144410 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:12:53.144607 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:12:53.144682 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:12:53.144856 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:12:53.144963 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:12:53.145023 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 21:12:53.145095 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 21:12:53.145114 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:12:53.146905 24264 raft_consensus.cc:1261] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Refusing update from remote peer 743ecfdb611c49aab54dff0e7018bccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:53.146957 24485 raft_consensus.cc:1261] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Refusing update from remote peer 743ecfdb611c49aab54dff0e7018bccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:12:53.147359 24116 consensus_queue.cc:1035] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:53.147532 24611 consensus_queue.cc:1035] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:12:53.150074 24620 mvcc.cc:205] Tried to move back new op lower bound from 7317079544406790144 to 7317079544295534592. Current Snapshot: MvccSnapshot[applied={T|T < 7317079544406790144}]
I20260810 21:12:53.150738 24621 mvcc.cc:205] Tried to move back new op lower bound from 7317079544406790144 to 7317079544295534592. Current Snapshot: MvccSnapshot[applied={T|T < 7317079544406790144}]
I20260810 21:12:53.150898 24622 mvcc.cc:205] Tried to move back new op lower bound from 7317079544406790144 to 7317079544295534592. Current Snapshot: MvccSnapshot[applied={T|T < 7317079544406790144 or (T in {7317079544406790144})}]
I20260810 21:12:53.152930 23772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:53.153040 23772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:12:53.153820 23772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Sending DeleteTablet for 3 replicas of tablet fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:12:53.154155 24234 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5bc92c43e94e3daa843d4127e260dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:53 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:59542
I20260810 21:12:53.154199 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 23849
I20260810 21:12:53.154225 24448 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5bc92c43e94e3daa843d4127e260dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:12:53 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:50022
I20260810 21:12:53.164176 24636 tablet_replica.cc:330] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: stopping tablet replica
I20260810 21:12:53.164351 24636 raft_consensus.cc:2229] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:53.164525 24636 raft_consensus.cc:2258] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:53.172631 24635 tablet_replica.cc:330] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: stopping tablet replica
I20260810 21:12:53.172839 24635 raft_consensus.cc:2229] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:12:53.172969 24635 raft_consensus.cc:2258] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:12:53.173074 24636 ts_tablet_manager.cc:1905] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:53.174324 24635 ts_tablet_manager.cc:1905] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:12:53.175395 24636 ts_tablet_manager.cc:1918] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:53.175469 24636 log.cc:1196] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:12:53.175760 24636 ts_tablet_manager.cc:1939] T fe5bc92c43e94e3daa843d4127e260dc P b0f1d25602354afd91ed47aa3ea506bd: Deleting consensus metadata
I20260810 21:12:53.175798 24635 ts_tablet_manager.cc:1918] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:12:53.175894 24635 log.cc:1196] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:12:53.176133 24635 ts_tablet_manager.cc:1939] T fe5bc92c43e94e3daa843d4127e260dc P 4a531952097d4f2d9b8f2dc4efd55eb5: Deleting consensus metadata
I20260810 21:12:53.176312 23749 catalog_manager.cc:4841] TS b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): tablet fe5bc92c43e94e3daa843d4127e260dc (table test-table [id=fc39c0feeb3147ba99189810db880409]) successfully deleted
I20260810 21:12:53.176556 23752 catalog_manager.cc:4841] TS 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): tablet fe5bc92c43e94e3daa843d4127e260dc (table test-table [id=fc39c0feeb3147ba99189810db880409]) successfully deleted
W20260810 21:12:53.179361 23755 connection.cc:535] client connection to 127.27.117.65:37733 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 21:12:53.179471 23755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 21:12:53.179677 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 24117
I20260810 21:12:53.179934 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:12:53.179809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:37733 (local address 127.0.0.1:43794)
0810 21:12:53.179862 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 21:12:53.179895 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 21:12:53.188064 23755 catalog_manager.cc:4584] TS 743ecfdb611c49aab54dff0e7018bccb (127.27.117.65:37733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe5bc92c43e94e3daa843d4127e260dc: Network error: Client connection negotiation failed: client connection to 127.27.117.65:37733: connect: Connection refused (error 111)
I20260810 21:12:53.199131 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 24336
I20260810 21:12:53.211359 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 23719
I20260810 21:13:13.220543 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45155
--webserver_interface=127.27.117.126
--webserver_port=34061
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:13.307264 24639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:13.307471 24639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:13.307497 24639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:13.308584 24639 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 21:13:13.308624 24639 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 21:13:13.308638 24639 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 21:13:13.308651 24639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:13.308666 24639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:13.308681 24639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:13.308698 24639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:13.308717 24639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:13.310340 24639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=34061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:13.310545 24639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:13.310711 24639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:13.315531 24656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.315510 24650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:13.315713 24639 server_base.cc:1034] running on GCE node
W20260810 21:13:13.315521 24647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:13.316025 24639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:13.316357 24639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:13.317531 24639 hybrid_clock.cc:648] HybridClock initialized: now 1786396393317512 us; error 31 us; skew 500 ppm
Aug 10 21:13:13 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:13.323599 24639 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:13:13.324069 24639 webserver.cc:460] Webserver started at http://127.27.117.126:34061/ using document root <none> and password file <none>
I20260810 21:13:13.324303 24639 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:13.324386 24639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:13.325832 24639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:13.326655 24666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.326870 24639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:13.326992 24639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "c1645a25315c47f1b501a78fe2a2ec6f"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "6d0e1713615dcec612b57cdf97e6871b"
server_key_iv: "4ee9f57855b1b204825a024ee274ef5a"
server_key_version: "encryptionkey@0"
I20260810 21:13:13.327359 24639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:13.350253 24639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:13.352222 24639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:13.352407 24639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:13.356990 24639 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:45155
I20260810 21:13:13.357023 24784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:45155 every 8 connection(s)
I20260810 21:13:13.357439 24639 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:13.358925 24789 sys_catalog.cc:263] Verifying existing consensus state
I20260810 21:13:13.359622 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap starting.
I20260810 21:13:13.367061 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 24639
I20260810 21:13:13.367399 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:37733
--local_ip_for_outbound_sockets=127.27.117.65
--tserver_master_addrs=127.27.117.126:45155
--webserver_port=41611
--webserver_interface=127.27.117.65
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:13:13.369618 24789 log.cc:824] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:13.371881 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:13:13.372196 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Bootstrap complete.
I20260810 21:13:13.374500 24789 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:13:13.374723 24789 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1645a25315c47f1b501a78fe2a2ec6f, State: Initialized, Role: FOLLOWER
I20260810 21:13:13.374881 24789 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:13:13.374951 24789 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:13.375012 24789 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:13.375072 24789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 2 FOLLOWER]: Advancing to term 3
I20260810 21:13:13.376030 24789 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:13:13.376132 24789 leader_election.cc:302] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1645a25315c47f1b501a78fe2a2ec6f; no voters:
I20260810 21:13:13.376334 24789 leader_election.cc:288] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 21:13:13.376370 24794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 3 FOLLOWER]: Leader election won for term 3
I20260810 21:13:13.376488 24794 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [term 3 LEADER]: Becoming Leader. State: Replica: c1645a25315c47f1b501a78fe2a2ec6f, State: Running, Role: LEADER
I20260810 21:13:13.376609 24789 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:13.376655 24794 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } }
I20260810 21:13:13.376951 24794 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1645a25315c47f1b501a78fe2a2ec6f. Latest consensus state: current_term: 3 leader_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } } }
I20260810 21:13:13.377027 24794 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:13.376979 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1645a25315c47f1b501a78fe2a2ec6f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45155 } } }
I20260810 21:13:13.377076 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:13.377306 24805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:13.378305 24805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:13.378405 24805 catalog_manager.cc:1259] Loaded cluster ID: 22555892b2dc4d9f87f29f4690311fa0
I20260810 21:13:13.378444 24805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:13.379547 24805 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:13.379721 24805 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Loaded TSK: 0
I20260810 21:13:13.379771 24805 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1645a25315c47f1b501a78fe2a2ec6f: Loaded TSK: 1
F20260810 21:13:13.380005 24805 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc3e7e089ad google::LogMessage::Fail() at ??:0
@ 0x7fc3e7e0b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc3e7e084de google::LogMessage::Flush() at ??:0
@ 0x7fc3e7e09a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc3ea575104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc3ea572d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc3efd621cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc3efd6248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc3efd62594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc3efd28c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc3efd32ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc3e9cd60d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc3e9ccb954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc3eab2f6db start_thread at ??:0
@ 0x7fc3e690661f clone at ??:0
W20260810 21:13:13.460915 24791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:13.461084 24791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:13.461107 24791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:13.462288 24791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:13.462348 24791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:13.462374 24791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:13.463959 24791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:37733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=41611
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:13.464174 24791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:13.464455 24791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:13.470443 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.470453 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.470422 24854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:13.470602 24791 server_base.cc:1034] running on GCE node
I20260810 21:13:13.471087 24791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:13.471369 24791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:13.472533 24791 hybrid_clock.cc:648] HybridClock initialized: now 1786396393472510 us; error 40 us; skew 500 ppm
Aug 10 21:13:13 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:13.475268 24791 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:13:13.475745 24791 webserver.cc:460] Webserver started at http://127.27.117.65:41611/ using document root <none> and password file <none>
I20260810 21:13:13.476007 24791 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:13.476075 24791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:13.477530 24791 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:13.478259 24883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.478454 24791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:13.478515 24791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "743ecfdb611c49aab54dff0e7018bccb"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "a632e0995c99986de9977d2dbf686b43"
server_key_iv: "79037ef7c7963ebaf12775c0854ea2af"
server_key_version: "encryptionkey@0"
I20260810 21:13:13.478843 24791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:13.493247 24791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:13.495510 24791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:13.495656 24791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:13.495898 24791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:13.496331 24894 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 21:13:13.498328 24791 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 21:13:13.498399 24791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 21:13:13.498443 24791 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 21:13:13.499043 24791 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 21:13:13.499101 24791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 21:13:13.499179 24894 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap starting.
I20260810 21:13:13.506737 24791 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:37733
I20260810 21:13:13.506830 25091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:37733 every 8 connection(s)
I20260810 21:13:13.507175 24791 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:13:13.510780 24894 log.cc:824] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:13.513461 24894 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 21:13:13.513649 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 24791
I20260810 21:13:13.513784 24894 tablet_bootstrap.cc:492] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Bootstrap complete.
I20260810 21:13:13.513945 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:45281
--local_ip_for_outbound_sockets=127.27.117.66
--tserver_master_addrs=127.27.117.126:45155
--webserver_port=36261
--webserver_interface=127.27.117.66
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:13:13.515178 24894 ts_tablet_manager.cc:1397] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260810 21:13:13.516552 24894 raft_consensus.cc:348] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:13.516880 24894 raft_consensus.cc:729] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 743ecfdb611c49aab54dff0e7018bccb, State: Initialized, Role: FOLLOWER
I20260810 21:13:13.517048 24894 consensus_queue.cc:260] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:13.517314 24894 ts_tablet_manager.cc:1428] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260810 21:13:13.586405 25097 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:13:13.507805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:55799)
0810 21:13:13.507980 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.508022 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:13.508437 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:13.586197 (+ 77760us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
W20260810 21:13:13.586689 25095 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:45155 (0 consecutive failures): Network error: Failed to ping master at 127.27.117.126:45155: Client connection negotiation failed: client connection to 127.27.117.126:45155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 21:13:13.587066 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.586936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:44453)
0810 21:13:13.586995 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.587035 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 21:13:13.587406 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.587309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:55203)
0810 21:13:13.587354 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.587379 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 21:13:13.587568 25095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:13:13.598196 25099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:13.598372 25099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:13.598397 25099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:13.599472 25099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:13.599505 25099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:13.599530 25099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:13.600952 25099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:45281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=36261
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:13.601133 25099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:13.601285 25099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:13.605983 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.606021 25107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.605990 25112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:13.606158 25099 server_base.cc:1034] running on GCE node
I20260810 21:13:13.606388 25099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:13.606654 25099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:13.607815 25099 hybrid_clock.cc:648] HybridClock initialized: now 1786396393607801 us; error 45 us; skew 500 ppm
Aug 10 21:13:13 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:13.610479 25099 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:13:13.610944 25099 webserver.cc:460] Webserver started at http://127.27.117.66:36261/ using document root <none> and password file <none>
I20260810 21:13:13.611137 25099 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:13.611178 25099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:13.612355 25099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:13.612949 25123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.613147 25099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:13:13.613220 25099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "4a531952097d4f2d9b8f2dc4efd55eb5"
format_stamp: "Formatted at 2026-08-10 21:12:31 on dist-test-slave-2kcd"
server_key: "716750832bbee9539879a3d76f01873e"
server_key_iv: "c976758ea1246f738f158cdb82a1cbb4"
server_key_version: "encryptionkey@0"
I20260810 21:13:13.613524 25099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:13.619527 25099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:13.621245 25099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:13.621387 25099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:13.621591 25099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:13.621865 25099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:13.621896 25099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.621946 25099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:13.621973 25099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.627951 25099 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:45281
I20260810 21:13:13.628000 25335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:45281 every 8 connection(s)
I20260810 21:13:13.628298 25099 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:13:13.629230 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.628904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:36317)
0810 21:13:13.629049 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.629111 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
W20260810 21:13:13.629635 25336 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:45155 (0 consecutive failures): Network error: Failed to ping master at 127.27.117.126:45155: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
I20260810 21:13:13.629765 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 25099
I20260810 21:13:13.629932 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.629819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:56797)
0810 21:13:13.629873 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.629907 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 21:13:13.630070 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:42855
--local_ip_for_outbound_sockets=127.27.117.67
--tserver_master_addrs=127.27.117.126:45155
--webserver_port=40961
--webserver_interface=127.27.117.67
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:13:13.630265 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.630169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:39659)
0810 21:13:13.630213 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.630236 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 21:13:13.630360 25336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 21:13:13.710307 25339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:13.710486 25339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:13.710510 25339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:13.711740 25339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:13.711781 25339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:13.711807 25339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:13.713276 25339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:42855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=40961
--tserver_master_addrs=127.27.117.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:13.713505 25339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:13.713729 25339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:13.718696 25361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.718719 25353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:13.718703 25355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:13.718899 25339 server_base.cc:1034] running on GCE node
I20260810 21:13:13.719162 25339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:13.719439 25339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:13.720618 25339 hybrid_clock.cc:648] HybridClock initialized: now 1786396393720600 us; error 35 us; skew 500 ppm
Aug 10 21:13:13 dist-test-slave-2kcd krb5kdc[22742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:13.723210 25339 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:13:13.723670 25339 webserver.cc:460] Webserver started at http://127.27.117.67:40961/ using document root <none> and password file <none>
I20260810 21:13:13.723917 25339 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:13.723987 25339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:13.725359 25339 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:13.725919 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.726101 25339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:13:13.726207 25339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "b0f1d25602354afd91ed47aa3ea506bd"
format_stamp: "Formatted at 2026-08-10 21:12:32 on dist-test-slave-2kcd"
server_key: "f692a993d63a74720bf625038f7aea9d"
server_key_iv: "bdaeb7973e6cd1f496e210b8a9cfc10f"
server_key_version: "encryptionkey@0"
I20260810 21:13:13.726506 25339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:13.749722 25339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:13.751976 25339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:13.752140 25339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:13.752363 25339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:13.752657 25339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:13.752710 25339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.752751 25339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:13.752799 25339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:13.758919 25339 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:42855
I20260810 21:13:13.758975 25572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:42855 every 8 connection(s)
I20260810 21:13:13.759289 25339 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:13.760274 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.759931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:49245)
0810 21:13:13.760078 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.760140 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260810 21:13:13.760699 25576 heartbeater.cc:644] Failed to heartbeat to 127.27.117.126:45155 (0 consecutive failures): Network error: Failed to ping master at 127.27.117.126:45155: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
I20260810 21:13:13.761058 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.760916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:42589)
0810 21:13:13.761006 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.761037 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 21:13:13.761384 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:13.761303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:50953)
0810 21:13:13.761336 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:13.761366 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260810 21:13:13.761509 25576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 21:13:13.765424 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 25339
I20260810 21:13:14.588227 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.587969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:53649)
0810 21:13:14.588121 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:14.588158 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260810 21:13:14.630964 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.630685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:41645)
0810 21:13:14.630827 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:14.630908 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260810 21:13:14.653416 25584 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:14.653591 25584 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:14.653965 25584 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
Aug 10 21:13:14 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.67@KRBTEST.COM
Aug 10 21:13:14 dist-test-slave-2kcd krb5kdc[22742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396393, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.66@KRBTEST.COM
I20260810 21:13:14.664453 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.654292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:41039 (local address 127.27.117.67:42855)
0810 21:13:14.654489 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:14.654499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:14.654542 (+ 43us) server_negotiation.cc:407] Connection header received
0810 21:13:14.654984 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:14.654986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:14.655037 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:14.655152 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:14.655967 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.656915 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.657915 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.658329 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.658455 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:14.660878 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:14.660882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:14.660882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:14.660892 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:14.662751 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:14.663227 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:14.663230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:14.663231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:14.663322 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:14.663620 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:14.663621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:14.663621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:14.663844 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:14.663937 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:14.664225 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 21:13:14.664361 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260810 21:13:14.664453 25580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.654146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42855 (local address 127.27.117.65:41039)
0810 21:13:14.654204 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:14.654216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:14.654278 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:14.655229 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:14.655232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:14.655663 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:14.655843 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:14.655849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.657046 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.657047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:14.657820 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:14.657823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.658435 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.658436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:14.658545 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:14.659189 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:14.659194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:14.660705 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:14.662866 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:14.662874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:14.662876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:14.663141 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:14.663410 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:14.663411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:14.663411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:14.663543 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:14.663920 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:14.663927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:14.664057 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 21:13:14.664159 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 21:13:14.664343 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":37}
I20260810 21:13:14.664565 25588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.654146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:45281 (local address 127.27.117.65:38759)
0810 21:13:14.654377 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:14.654387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:14.654429 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:14.655279 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:14.655280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:14.655663 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:14.655860 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:14.655862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.656987 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.656989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:14.657820 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:14.657822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.658432 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.658433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:14.658547 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:14.659131 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:14.659139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:14.661114 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:14.662946 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:14.662947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:14.662947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:14.663156 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:14.663410 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:14.663411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:14.663411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:14.663546 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:14.663930 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:14.663931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:14.664058 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 21:13:14.664159 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 21:13:14.664422 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":210,"thread_start_us":57,"threads_started":1}
I20260810 21:13:14.664487 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.654286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:38759 (local address 127.27.117.66:45281)
0810 21:13:14.654604 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:14.654616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:14.654649 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:13:14.655079 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:14.655081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:14.655142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:14.655282 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:14.655992 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.656892 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.657955 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:14.658332 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:14.658465 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:14.661230 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:14.661233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:14.661233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:14.661242 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:14.662845 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:14.663238 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:14.663241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:14.663242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:14.663330 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:14.663616 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:14.663617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:14.663618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:14.663841 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:14.663936 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:14.664227 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 21:13:14.664372 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":60,"threads_started":1}
W20260810 21:13:14.665444 24884 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
W20260810 21:13:14.665787 24888 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:13:14.665845 24888 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb; no voters: 4a531952097d4f2d9b8f2dc4efd55eb5, b0f1d25602354afd91ed47aa3ea506bd
I20260810 21:13:14.665951 25584 raft_consensus.cc:2726] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:13:14.761986 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:14.761781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:41047)
0810 21:13:14.761907 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:14.761952 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260810 21:13:15.589084 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:15.588846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:48367)
0810 21:13:15.588990 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:15.589026 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260810 21:13:15.631645 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:15.631446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:39305)
0810 21:13:15.631572 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:15.631609 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260810 21:13:15.762682 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:15.762492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:46463)
0810 21:13:15.762616 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:15.762649 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260810 21:13:16.304625 25603 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:16.304729 25603 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:16.304996 25603 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
W20260810 21:13:16.305399 24888 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
W20260810 21:13:16.305480 24884 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:13:16.305542 24888 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb; no voters: 4a531952097d4f2d9b8f2dc4efd55eb5, b0f1d25602354afd91ed47aa3ea506bd
I20260810 21:13:16.305635 25603 raft_consensus.cc:2726] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:13:16.589756 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:16.589537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:51777)
0810 21:13:16.589672 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:16.589708 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260810 21:13:16.632344 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:16.632101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:59363)
0810 21:13:16.632258 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:16.632304 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260810 21:13:16.763478 25607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:16.763256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:50885)
0810 21:13:16.763404 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:16.763439 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260810 21:13:17.590440 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:17.590233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:39081)
0810 21:13:17.590355 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:17.590392 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260810 21:13:17.633195 25611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:17.632772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:58311)
0810 21:13:17.633123 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:17.633159 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":161,"threads_started":1}
I20260810 21:13:17.764142 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:17.763926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:60317)
0810 21:13:17.764066 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:17.764103 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260810 21:13:18.047299 25614 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:18.047402 25614 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:18.047575 25614 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
W20260810 21:13:18.047986 24884 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
W20260810 21:13:18.048089 24888 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:13:18.048151 24888 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb; no voters: 4a531952097d4f2d9b8f2dc4efd55eb5, b0f1d25602354afd91ed47aa3ea506bd
I20260810 21:13:18.048221 25614 raft_consensus.cc:2726] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:13:18.591194 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:18.590984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:43565)
0810 21:13:18.591112 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:18.591147 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260810 21:13:18.633848 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:18.633611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:51761)
0810 21:13:18.633742 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:18.633783 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260810 21:13:18.764914 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:18.764606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:33915)
0810 21:13:18.764726 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:18.764763 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260810 21:13:19.591910 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:19.591699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:50371)
0810 21:13:19.591827 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:19.591863 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260810 21:13:19.634603 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:19.634374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:37071)
0810 21:13:19.634520 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:19.634557 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260810 21:13:19.675413 25625 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:19.675514 25625 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:19.675695 25625 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
W20260810 21:13:19.676112 24884 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
W20260810 21:13:19.676180 24888 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:13:19.676209 24884 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb; no voters: 4a531952097d4f2d9b8f2dc4efd55eb5, b0f1d25602354afd91ed47aa3ea506bd
I20260810 21:13:19.676280 25625 raft_consensus.cc:2726] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:13:19.765618 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:19.765390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:56133)
0810 21:13:19.765522 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:19.765561 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260810 21:13:20.592582 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:20.592367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:43493)
0810 21:13:20.592498 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:20.592536 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260810 21:13:20.635290 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:20.635088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:54167)
0810 21:13:20.635221 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:20.635254 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260810 21:13:20.766350 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:20.766159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:48175)
0810 21:13:20.766278 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:20.766312 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260810 21:13:21.593248 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:21.593046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:48185)
0810 21:13:21.593170 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:21.593202 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260810 21:13:21.635973 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:21.635741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:45103)
0810 21:13:21.635887 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:21.635934 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260810 21:13:21.767102 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:21.766868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:45719)
0810 21:13:21.767028 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:21.767064 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260810 21:13:22.593971 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:22.593720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:55693)
0810 21:13:22.593869 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:22.593918 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260810 21:13:22.636677 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:22.636478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:44085)
0810 21:13:22.636601 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:22.636632 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260810 21:13:22.767729 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:22.767527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:35391)
0810 21:13:22.767656 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:22.767689 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260810 21:13:23.594939 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:23.594633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:54355)
0810 21:13:23.594826 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:23.594867 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260810 21:13:23.637379 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:23.637177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:47289)
0810 21:13:23.637298 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:23.637335 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260810 21:13:23.768374 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:23.768171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:60243)
0810 21:13:23.768294 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:23.768329 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260810 21:13:24.595713 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:24.595462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:56315)
0810 21:13:24.595623 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:24.595661 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260810 21:13:24.638144 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:24.637941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:52763)
0810 21:13:24.638065 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:24.638101 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260810 21:13:24.769142 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:24.768942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:37153)
0810 21:13:24.769068 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:24.769102 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260810 21:13:25.596462 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:25.596215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:42337)
0810 21:13:25.596377 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:25.596414 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260810 21:13:25.638784 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:25.638573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:34853)
0810 21:13:25.638709 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:25.638744 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260810 21:13:25.769852 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:25.769574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:57745)
0810 21:13:25.769738 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:25.769784 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260810 21:13:25.978016 25664 raft_consensus.cc:482] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:25.978116 25664 raft_consensus.cc:504] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f1d25602354afd91ed47aa3ea506bd" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42855 } } peers { permanent_uuid: "4a531952097d4f2d9b8f2dc4efd55eb5" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 45281 } } peers { permanent_uuid: "743ecfdb611c49aab54dff0e7018bccb" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 37733 } }
I20260810 21:13:25.978276 25664 leader_election.cc:288] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855), 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281)
W20260810 21:13:25.978703 24884 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0f1d25602354afd91ed47aa3ea506bd (127.27.117.67:42855): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
W20260810 21:13:25.978811 24888 leader_election.cc:341] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a531952097d4f2d9b8f2dc4efd55eb5 (127.27.117.66:45281): Not found: Tablet not found: fe5bc92c43e94e3daa843d4127e260dc
I20260810 21:13:25.978874 24888 leader_election.cc:302] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 743ecfdb611c49aab54dff0e7018bccb; no voters: 4a531952097d4f2d9b8f2dc4efd55eb5, b0f1d25602354afd91ed47aa3ea506bd
I20260810 21:13:25.978967 25664 raft_consensus.cc:2726] T fe5bc92c43e94e3daa843d4127e260dc P 743ecfdb611c49aab54dff0e7018bccb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 21:13:26.597126 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:26.596925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:55267)
0810 21:13:26.597047 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:26.597078 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260810 21:13:26.639408 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:26.639200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:52477)
0810 21:13:26.639323 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:26.639361 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260810 21:13:26.770658 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:26.770437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:53639)
0810 21:13:26.770574 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:26.770610 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260810 21:13:27.597918 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:27.597599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:34721)
0810 21:13:27.597771 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:27.597863 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260810 21:13:27.640215 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:27.639962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:34839)
0810 21:13:27.640104 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:27.640164 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260810 21:13:27.771359 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:27.771150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:43373)
0810 21:13:27.771279 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:27.771316 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260810 21:13:28.598783 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:28.598566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.65:57001)
0810 21:13:28.598700 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:28.598737 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260810 21:13:28.641196 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:28.640929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.66:60607)
0810 21:13:28.641100 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:28.641142 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260810 21:13:28.766290 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 24791
I20260810 21:13:28.773345 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:28.773012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:45155 (local address 127.27.117.67:58711)
0810 21:13:28.773271 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:28.773307 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.126:45155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":59,"threads_started":1}
I20260810 21:13:28.777931 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 25099
I20260810 21:13:28.788563 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 25339
2026-08-10T21:13:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58630 ms)
[----------] 30 tests from SecurityITest (241910 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:28 dist-test-slave-2kcd krb5kdc[25683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:28 dist-test-slave-2kcd krb5kdc[25683](info): set up 2 sockets
Aug 10 21:13:28 dist-test-slave-2kcd krb5kdc[25683](info): commencing operation
krb5kdc: starting...
W20260810 21:13:30.079066 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:13:30.199800 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38273
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:30.283999 25701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:30.284225 25701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:30.284263 25701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:30.285457 25701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:30.285523 25701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:30.285547 25701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:30.285578 25701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:30.285609 25701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:30.287045 25701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:30.287312 25701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:30.287537 25701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:30.292547 25714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.292564 25710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.292524 25709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:30.292572 25701 server_base.cc:1034] running on GCE node
I20260810 21:13:30.293030 25701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:30.293300 25701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:30.294441 25701 hybrid_clock.cc:648] HybridClock initialized: now 1786396410294433 us; error 33 us; skew 500 ppm
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:30.303799 25701 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:13:30.304519 25701 webserver.cc:460] Webserver started at http://127.27.117.126:40657/ using document root <none> and password file <none>
I20260810 21:13:30.305110 25701 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:30.305188 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:30.305357 25701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:30.307212 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a0277e39c7984c61bd09e4413828d676"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "425edfeb7e5e1e9d25ccfd440578bdf3"
server_key_iv: "5da8ec8bedaec4df431ebc14914908fc"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.307960 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a0277e39c7984c61bd09e4413828d676"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "425edfeb7e5e1e9d25ccfd440578bdf3"
server_key_iv: "5da8ec8bedaec4df431ebc14914908fc"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.309852 25701 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.005s
I20260810 21:13:30.310899 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.311106 25701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:30.311288 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a0277e39c7984c61bd09e4413828d676"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "425edfeb7e5e1e9d25ccfd440578bdf3"
server_key_iv: "5da8ec8bedaec4df431ebc14914908fc"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.311465 25701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:30.337241 25701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:30.341614 25701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:30.341776 25701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:30.346655 25701 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:38273
I20260810 21:13:30.346691 25829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:38273 every 8 connection(s)
I20260810 21:13:30.347051 25701 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:30.347980 25832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:30.350433 25832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: Bootstrap starting.
I20260810 21:13:30.351023 25832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:30.351306 25832 log.cc:824] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:30.352007 25832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: No bootstrap required, opened a new log
I20260810 21:13:30.353739 25832 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0277e39c7984c61bd09e4413828d676" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38273 } }
I20260810 21:13:30.353894 25832 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:30.353942 25832 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0277e39c7984c61bd09e4413828d676, State: Initialized, Role: FOLLOWER
I20260810 21:13:30.354094 25832 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0277e39c7984c61bd09e4413828d676" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38273 } }
I20260810 21:13:30.354157 25832 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:30.354198 25832 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:30.354249 25832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:30.354682 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 25701
I20260810 21:13:30.354772 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:30.354807 25832 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0277e39c7984c61bd09e4413828d676" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38273 } }
I20260810 21:13:30.354945 25832 leader_election.cc:302] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0277e39c7984c61bd09e4413828d676; no voters:
I20260810 21:13:30.354995 28117 external_mini_cluster.cc:1428] Setting key 6874f5c1547434b70fe6d76e2f5297d9
I20260810 21:13:30.355144 25832 leader_election.cc:288] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:30.355163 25835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:30.355357 25835 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [term 1 LEADER]: Becoming Leader. State: Replica: a0277e39c7984c61bd09e4413828d676, State: Running, Role: LEADER
I20260810 21:13:30.355418 25832 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:30.356086 25835 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0277e39c7984c61bd09e4413828d676" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38273 } }
I20260810 21:13:30.356691 25837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0277e39c7984c61bd09e4413828d676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0277e39c7984c61bd09e4413828d676" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38273 } } }
I20260810 21:13:30.356849 25837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:30.356959 25839 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0277e39c7984c61bd09e4413828d676. Latest consensus state: current_term: 1 leader_uuid: "a0277e39c7984c61bd09e4413828d676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0277e39c7984c61bd09e4413828d676" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38273 } } }
I20260810 21:13:30.357224 25839 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676 [sys.catalog]: This master's current role is: LEADER
W20260810 21:13:30.357625 25864 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:13:30.357689 25864 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:13:30.357756 25866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:30.357960 25866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:30.359395 25866 catalog_manager.cc:1347] Generated new cluster ID: 0d14c50641b04d97a876d51b6708da46
I20260810 21:13:30.359455 25866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:30.367094 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.355461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.27.117.126:38273)
0810 21:13:30.356002 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.356014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.356050 (+ 36us) server_negotiation.cc:407] Connection header received
0810 21:13:30.356193 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.356201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.356271 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.356413 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:30.358081 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.358999 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.360086 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.360484 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.360567 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.363406 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:30.363409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:30.363410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:30.363421 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:30.365380 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.365844 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.365847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.365847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.365955 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.366193 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.366194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.366194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.366469 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:30.366631 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.366732 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.366859 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":71,"threads_started":1}
I20260810 21:13:30.389024 25866 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:30.389755 25866 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:30.394842 25866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: Generated new TSK 0
I20260810 21:13:30.395021 25866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:13:30.423596 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38273
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:30.505033 25885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:30.505223 25885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:30.505249 25885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:30.506369 25885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:30.506413 25885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:30.506438 25885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:30.508008 25885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:30.508252 25885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:30.508476 25885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:13:30.513502 25885 server_base.cc:1034] running on GCE node
W20260810 21:13:30.513458 25897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.513458 25896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.513464 25904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:30.513914 25885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:30.514173 25885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:30.515336 25885 hybrid_clock.cc:648] HybridClock initialized: now 1786396410515331 us; error 36 us; skew 500 ppm
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:30.518031 25885 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:13:30.518440 25885 webserver.cc:460] Webserver started at http://127.27.117.65:41897/ using document root <none> and password file <none>
I20260810 21:13:30.518668 25885 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:30.518711 25885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:30.518884 25885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:30.519861 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "a38c19d8d4fc4d82920addbcede9c257"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "637c567d5d1fb19ae65374bb8716a346"
server_key_iv: "17c53826a8f017a2fc92958373d19585"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.520213 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "a38c19d8d4fc4d82920addbcede9c257"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "637c567d5d1fb19ae65374bb8716a346"
server_key_iv: "17c53826a8f017a2fc92958373d19585"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.521526 25885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:30.522235 25915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.522400 25885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:30.522507 25885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "a38c19d8d4fc4d82920addbcede9c257"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "637c567d5d1fb19ae65374bb8716a346"
server_key_iv: "17c53826a8f017a2fc92958373d19585"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.522578 25885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:30.531104 25885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:30.532860 25885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:30.533012 25885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:30.533254 25885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:30.533547 25885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:30.533602 25885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.533643 25885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:30.533669 25885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.539593 25885 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:41301
I20260810 21:13:30.539664 26130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:41301 every 8 connection(s)
I20260810 21:13:30.539980 25885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:30.548286 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 25885
I20260810 21:13:30.548430 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:13:30.548656 28117 external_mini_cluster.cc:1428] Setting key 49567c5777359bb0cc795e91ad3c896c
I20260810 21:13:30.549235 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.540720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:37803 (local address 127.27.117.126:38273)
0810 21:13:30.540761 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.540764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.540970 (+ 206us) server_negotiation.cc:407] Connection header received
0810 21:13:30.541473 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.541475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.541490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.541550 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:30.542468 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.542970 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.543965 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.544183 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.544262 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.546545 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:30.546546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:30.546546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:30.546553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:30.547737 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.548151 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.548152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.548153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.548215 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.548499 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.548500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.548501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.548629 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:30.548705 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.549050 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.549147 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 21:13:30.549392 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.540676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38273 (local address 127.27.117.65:37803)
0810 21:13:30.540845 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.540890 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.541367 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.541616 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.541620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.542189 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:30.542365 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.542369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.543078 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.543079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.543866 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.543869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.544272 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.544273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.544401 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.544973 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:30.544981 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:30.546432 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:30.547836 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:30.547844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:30.547846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:30.548070 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:30.548287 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:30.548288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:30.548288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:30.548434 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:30.548707 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:30.548715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:30.548907 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.549018 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.549173 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260810 21:13:30.549927 26131 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38273
I20260810 21:13:30.550040 26131 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:30.550210 26131 heartbeater.cc:511] Master 127.27.117.126:38273 requested a full tablet report, sending...
I20260810 21:13:30.550748 25763 ts_manager.cc:194] Registered new tserver with Master: a38c19d8d4fc4d82920addbcede9c257 (127.27.117.65:41301)
I20260810 21:13:30.551625 25763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:37803
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:13:30.575060 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38273
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:30.659581 26138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:30.659837 26138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:30.659874 26138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:30.661024 26138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:30.661087 26138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:30.661129 26138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:30.662572 26138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:30.662822 26138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:30.663048 26138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:30.668181 26150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.668188 26153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.668254 26148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:30.668586 26138 server_base.cc:1034] running on GCE node
I20260810 21:13:30.668777 26138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:30.669024 26138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:30.670188 26138 hybrid_clock.cc:648] HybridClock initialized: now 1786396410670178 us; error 29 us; skew 500 ppm
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:30.672611 26138 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:13:30.673089 26138 webserver.cc:460] Webserver started at http://127.27.117.66:41961/ using document root <none> and password file <none>
I20260810 21:13:30.673290 26138 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:30.673331 26138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:30.673421 26138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:30.674365 26138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "472fcac0afc041098fe095e9fef56953"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "18c36d27d21afd2a6e8110cde6e8a491"
server_key_iv: "9ef1f60e60c2dfc4152650d3d70890ac"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.674664 26138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "472fcac0afc041098fe095e9fef56953"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "18c36d27d21afd2a6e8110cde6e8a491"
server_key_iv: "9ef1f60e60c2dfc4152650d3d70890ac"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.675894 26138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:30.676590 26165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.676797 26138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:30.676878 26138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "472fcac0afc041098fe095e9fef56953"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "18c36d27d21afd2a6e8110cde6e8a491"
server_key_iv: "9ef1f60e60c2dfc4152650d3d70890ac"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.676934 26138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:30.697683 26138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:30.700073 26138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:30.700218 26138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:30.700457 26138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:30.700851 26138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:30.700891 26138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.700912 26138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:30.700949 26138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.706825 26138 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:33255
I20260810 21:13:30.706923 26368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:33255 every 8 connection(s)
I20260810 21:13:30.707244 26138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:13:30.710531 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 26138
I20260810 21:13:30.710649 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:13:30.710853 28117 external_mini_cluster.cc:1428] Setting key 32e9470df830d70044ab3ae7ccc28ebb
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:30.718379 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.708007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38273 (local address 127.27.117.66:42005)
0810 21:13:30.708162 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.708205 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.708714 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.709003 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.709007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.709568 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:30.709765 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.709769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.710566 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.710567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.711789 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.711793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.712165 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.712166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.712315 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.712883 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:30.712893 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:30.714975 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:30.716946 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:30.716954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:30.716956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:30.717286 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:30.717513 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:30.717514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:30.717514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:30.717610 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:30.717901 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:30.717906 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:30.718016 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.718098 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.718215 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260810 21:13:30.718381 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.708046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:42005 (local address 127.27.117.126:38273)
0810 21:13:30.708107 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.708109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.708293 (+ 184us) server_negotiation.cc:407] Connection header received
0810 21:13:30.708844 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.708845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.708855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.708943 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:30.709911 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.710464 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.711884 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.712083 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.712132 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.715092 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:30.715094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:30.715094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:30.715119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:30.716813 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.717357 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.717359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.717359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.717440 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.717677 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.717678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.717679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.717833 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:30.717908 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.718150 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.718250 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 21:13:30.718997 26372 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38273
I20260810 21:13:30.719094 26372 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:30.719292 26372 heartbeater.cc:511] Master 127.27.117.126:38273 requested a full tablet report, sending...
I20260810 21:13:30.719683 25763 ts_manager.cc:194] Registered new tserver with Master: 472fcac0afc041098fe095e9fef56953 (127.27.117.66:33255)
I20260810 21:13:30.720062 25763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:42005
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:13:30.739655 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38273
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:38463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:30.821719 26384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:30.821949 26384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:30.821990 26384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:30.823133 26384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:30.823201 26384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:30.823248 26384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:30.824892 26384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:38463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:30.825147 26384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:30.825378 26384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:30.830981 26392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.831004 26396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:30.831012 26394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:30.831175 26384 server_base.cc:1034] running on GCE node
I20260810 21:13:30.831357 26384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:30.831547 26384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:30.832683 26384 hybrid_clock.cc:648] HybridClock initialized: now 1786396410832672 us; error 27 us; skew 500 ppm
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:30.835141 26384 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:13:30.835523 26384 webserver.cc:460] Webserver started at http://127.27.117.67:36723/ using document root <none> and password file <none>
I20260810 21:13:30.835701 26384 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:30.835742 26384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:30.835829 26384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:30.836889 26384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "3663edb7aca246be93ac41c8b151ef9e"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "f703eb5d77dcedbc47ca197cce37cbf4"
server_key_iv: "ec3e10ff2729435125a96b942b8fc343"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.837210 26384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "3663edb7aca246be93ac41c8b151ef9e"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "f703eb5d77dcedbc47ca197cce37cbf4"
server_key_iv: "ec3e10ff2729435125a96b942b8fc343"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.838562 26384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:13:30.839241 26412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.839418 26384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.839527 26384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "3663edb7aca246be93ac41c8b151ef9e"
format_stamp: "Formatted at 2026-08-10 21:13:30 on dist-test-slave-2kcd"
server_key: "f703eb5d77dcedbc47ca197cce37cbf4"
server_key_iv: "ec3e10ff2729435125a96b942b8fc343"
server_key_version: "encryptionkey@0"
I20260810 21:13:30.839599 26384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:30.850005 26384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:30.851722 26384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:30.851883 26384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:30.852111 26384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:30.852439 26384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:30.852495 26384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.852535 26384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:30.852566 26384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:30.859134 26384 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:40841
I20260810 21:13:30.859213 26621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:40841 every 8 connection(s)
I20260810 21:13:30.859539 26384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:30.865089 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 26384
I20260810 21:13:30.865222 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
Aug 10 21:13:30 dist-test-slave-2kcd krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:30.865463 28117 external_mini_cluster.cc:1428] Setting key dd29c1775df6c7966de03356e41de1de
I20260810 21:13:30.868620 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.860271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:46681 (local address 127.27.117.126:38273)
0810 21:13:30.860332 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.860336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.860557 (+ 221us) server_negotiation.cc:407] Connection header received
0810 21:13:30.861132 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.861133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.861143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.861198 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:30.862108 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.862595 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.863529 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.863708 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.863757 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.865939 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:30.865941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:30.865941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:30.865948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:30.867243 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.867641 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.867643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.867643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.867703 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.867966 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.867968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.867968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.868083 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:30.868160 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.868434 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.868520 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:13:30.868808 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.860260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38273 (local address 127.27.117.67:46681)
0810 21:13:30.860431 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.860472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.861046 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.861249 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.861252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.861838 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:30.862032 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.862036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.862676 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.862677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.863450 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.863452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.863780 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.863781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.863874 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.864300 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:30.864308 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:30.865841 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:30.867327 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:30.867334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:30.867336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:30.867560 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:30.867773 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:30.867774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:30.867774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:30.867887 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:30.868150 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:30.868157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:30.868293 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.868396 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.868562 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260810 21:13:30.869270 26623 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38273
I20260810 21:13:30.869349 26623 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:30.869544 26623 heartbeater.cc:511] Master 127.27.117.126:38273 requested a full tablet report, sending...
I20260810 21:13:30.869910 25763 ts_manager.cc:194] Registered new tserver with Master: 3663edb7aca246be93ac41c8b151ef9e (127.27.117.67:40841)
I20260810 21:13:30.870283 25763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:46681
I20260810 21:13:30.877317 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:13:30.884069 25841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.878639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.27.117.126:38273)
0810 21:13:30.878699 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.878702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.878804 (+ 102us) server_negotiation.cc:407] Connection header received
0810 21:13:30.878841 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.878842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.878850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.878895 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:30.879379 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.879936 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.880677 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.880886 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.880949 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.881952 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:30.881953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:30.881953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:30.881959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:30.883118 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.883426 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.883426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.883427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.883475 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:30.883638 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:30.883639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:30.883639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:30.883762 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:30.883834 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.883927 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.883982 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 21:13:30.885505 25763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:13:30.887082 25763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:13:30.893608 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.889467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.27.117.65:41301)
0810 21:13:30.889688 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.889692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.890102 (+ 410us) server_negotiation.cc:407] Connection header received
0810 21:13:30.890152 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.890155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.890172 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.890252 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:30.890401 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.891479 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.892621 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.893178 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.893262 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.893320 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.893476 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.893515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
I20260810 21:13:30.893625 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.889218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:40841 (local address 127.0.0.1:33358)
0810 21:13:30.889669 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.889679 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.889717 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.889967 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.889968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.889971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:30.890016 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.890018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.891082 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.891083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.892378 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.892382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.893352 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.893353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.893429 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.893434 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.893468 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.893521 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":6,"thread_start_us":72,"threads_started":1}
I20260810 21:13:30.893642 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.889173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:33255 (local address 127.0.0.1:60932)
0810 21:13:30.889469 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.889511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.889579 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.889925 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.889928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.889931 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:30.890011 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.890014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.891338 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.891339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.892380 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.892382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.893350 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.893351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.893435 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.893439 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.893468 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.893524 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":22,"thread_start_us":64,"threads_started":1}
I20260810 21:13:30.893808 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.889455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.27.117.66:33255)
0810 21:13:30.889692 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.889697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.889707 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:30.889747 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.889750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.889767 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.889850 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:30.890106 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.891229 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.892487 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.893240 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.893568 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.893636 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.893663 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.893711 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
I20260810 21:13:30.894032 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.889218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:41301 (local address 127.0.0.1:49324)
0810 21:13:30.889781 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.889791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.889866 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.890263 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.890264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.890266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:30.890309 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.890310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.891581 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.891582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.892526 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.892530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.893264 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.893266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.893344 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.893358 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.893456 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.893509 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260810 21:13:30.894309 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.889450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.27.117.67:40841)
0810 21:13:30.889753 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.889756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.889763 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:13:30.889792 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.889793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.889804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.889970 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:30.890097 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.890973 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.892485 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.893239 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.894105 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.894170 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.894195 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.894233 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
I20260810 21:13:30.894541 26249 tablet_service.cc:1467] Processing CreateTablet for tablet 03da8abac27d4765abc37989ebdcd67b (DEFAULT_TABLE table=test-table [id=eb7b93206e094d0bbb743f7729b29178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:30.894779 26005 tablet_service.cc:1467] Processing CreateTablet for tablet 03da8abac27d4765abc37989ebdcd67b (DEFAULT_TABLE table=test-table [id=eb7b93206e094d0bbb743f7729b29178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:30.894901 26249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03da8abac27d4765abc37989ebdcd67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:30.894874 26505 tablet_service.cc:1467] Processing CreateTablet for tablet 03da8abac27d4765abc37989ebdcd67b (DEFAULT_TABLE table=test-table [id=eb7b93206e094d0bbb743f7729b29178]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:30.895051 26005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03da8abac27d4765abc37989ebdcd67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:30.895185 26505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03da8abac27d4765abc37989ebdcd67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:30.897841 26654 tablet_bootstrap.cc:492] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: Bootstrap starting.
I20260810 21:13:30.898295 26655 tablet_bootstrap.cc:492] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Bootstrap starting.
I20260810 21:13:30.898535 26654 tablet_bootstrap.cc:654] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:30.899080 26655 tablet_bootstrap.cc:654] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:30.899147 26654 log.cc:824] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:30.899624 26655 log.cc:824] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:30.899996 26656 tablet_bootstrap.cc:492] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Bootstrap starting.
I20260810 21:13:30.900044 26654 tablet_bootstrap.cc:492] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: No bootstrap required, opened a new log
I20260810 21:13:30.900098 26654 ts_tablet_manager.cc:1397] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:30.900401 26655 tablet_bootstrap.cc:492] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: No bootstrap required, opened a new log
I20260810 21:13:30.900496 26655 ts_tablet_manager.cc:1397] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:30.900632 26656 tablet_bootstrap.cc:654] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:30.901221 26656 log.cc:824] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:30.902057 26656 tablet_bootstrap.cc:492] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: No bootstrap required, opened a new log
I20260810 21:13:30.902129 26656 ts_tablet_manager.cc:1397] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 21:13:30.902360 26655 raft_consensus.cc:348] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.902518 26655 raft_consensus.cc:374] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:30.902563 26655 raft_consensus.cc:729] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3663edb7aca246be93ac41c8b151ef9e, State: Initialized, Role: FOLLOWER
I20260810 21:13:30.902456 26654 raft_consensus.cc:348] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.902617 26654 raft_consensus.cc:374] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:30.902686 26654 raft_consensus.cc:729] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38c19d8d4fc4d82920addbcede9c257, State: Initialized, Role: FOLLOWER
I20260810 21:13:30.902669 26655 consensus_queue.cc:260] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.902815 26654 consensus_queue.cc:260] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.902957 26655 ts_tablet_manager.cc:1428] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 21:13:30.903077 26623 heartbeater.cc:503] Master 127.27.117.126:38273 was elected leader, sending a full tablet report...
I20260810 21:13:30.903097 26654 ts_tablet_manager.cc:1428] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:30.903249 26131 heartbeater.cc:503] Master 127.27.117.126:38273 was elected leader, sending a full tablet report...
I20260810 21:13:30.904170 26656 raft_consensus.cc:348] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.904335 26656 raft_consensus.cc:374] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:30.904372 26656 raft_consensus.cc:729] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472fcac0afc041098fe095e9fef56953, State: Initialized, Role: FOLLOWER
I20260810 21:13:30.904470 26656 consensus_queue.cc:260] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.904687 26656 ts_tablet_manager.cc:1428] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:13:30.904767 26372 heartbeater.cc:503] Master 127.27.117.126:38273 was elected leader, sending a full tablet report...
W20260810 21:13:30.957803 26375 tablet.cc:2369] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:30.984823 26663 raft_consensus.cc:482] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:30.984941 26663 raft_consensus.cc:504] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.985380 26663 leader_election.cc:288] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 472fcac0afc041098fe095e9fef56953 (127.27.117.66:33255), 3663edb7aca246be93ac41c8b151ef9e (127.27.117.67:40841)
I20260810 21:13:30.989352 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.985690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:39513 (local address 127.27.117.67:40841)
0810 21:13:30.985735 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.985737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.985747 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:30.985775 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.985775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.985793 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.985843 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:30.986077 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.986847 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.988354 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.988982 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.989061 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.989070 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.989244 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.989283 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:13:30.989352 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.985656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:53879 (local address 127.27.117.66:33255)
0810 21:13:30.985713 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:30.985715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:30.985922 (+ 207us) server_negotiation.cc:407] Connection header received
0810 21:13:30.985946 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:30.985947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:30.985955 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:30.985998 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:30.986170 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.986854 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.988354 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.988986 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.989061 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.989070 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:30.989244 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 21:13:30.989283 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:13:30.989362 26133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.985549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:40841 (local address 127.27.117.65:39513)
0810 21:13:30.985618 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.985634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.985691 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.985875 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.985876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.985878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:30.985972 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.985974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.986939 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.986940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.988255 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.988259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.989073 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.989074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.989157 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.989159 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.989219 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.989267 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":49}
I20260810 21:13:30.989370 26668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:30.985549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:33255 (local address 127.27.117.65:53879)
0810 21:13:30.985795 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:30.985811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:30.985887 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:30.986029 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:30.986030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:30.986031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:30.986094 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.986096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.986946 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.986947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.988255 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:30.988259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:30.989073 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:30.989074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:30.989152 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:30.989157 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:13:30.989219 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 21:13:30.989267 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":144,"thread_start_us":116,"threads_started":1}
I20260810 21:13:30.989782 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03da8abac27d4765abc37989ebdcd67b" candidate_uuid: "a38c19d8d4fc4d82920addbcede9c257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3663edb7aca246be93ac41c8b151ef9e" is_pre_election: true
I20260810 21:13:30.989954 26538 raft_consensus.cc:2454] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a38c19d8d4fc4d82920addbcede9c257 in term 0.
I20260810 21:13:30.989984 26283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03da8abac27d4765abc37989ebdcd67b" candidate_uuid: "a38c19d8d4fc4d82920addbcede9c257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472fcac0afc041098fe095e9fef56953" is_pre_election: true
I20260810 21:13:30.990141 26283 raft_consensus.cc:2454] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a38c19d8d4fc4d82920addbcede9c257 in term 0.
I20260810 21:13:30.990199 25918 leader_election.cc:302] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3663edb7aca246be93ac41c8b151ef9e, a38c19d8d4fc4d82920addbcede9c257; no voters:
I20260810 21:13:30.990338 26663 raft_consensus.cc:2781] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:13:30.990410 26663 raft_consensus.cc:482] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:13:30.990442 26663 raft_consensus.cc:3037] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:30.991174 26663 raft_consensus.cc:504] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.991323 26663 leader_election.cc:288] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [CANDIDATE]: Term 1 election: Requested vote from peers 472fcac0afc041098fe095e9fef56953 (127.27.117.66:33255), 3663edb7aca246be93ac41c8b151ef9e (127.27.117.67:40841)
I20260810 21:13:30.991466 26283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03da8abac27d4765abc37989ebdcd67b" candidate_uuid: "a38c19d8d4fc4d82920addbcede9c257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472fcac0afc041098fe095e9fef56953"
I20260810 21:13:30.991487 26538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03da8abac27d4765abc37989ebdcd67b" candidate_uuid: "a38c19d8d4fc4d82920addbcede9c257" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3663edb7aca246be93ac41c8b151ef9e"
I20260810 21:13:30.991552 26283 raft_consensus.cc:3037] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:30.991559 26538 raft_consensus.cc:3037] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:30.992450 26538 raft_consensus.cc:2454] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a38c19d8d4fc4d82920addbcede9c257 in term 1.
I20260810 21:13:30.992450 26283 raft_consensus.cc:2454] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a38c19d8d4fc4d82920addbcede9c257 in term 1.
I20260810 21:13:30.992677 25917 leader_election.cc:302] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472fcac0afc041098fe095e9fef56953, a38c19d8d4fc4d82920addbcede9c257; no voters:
I20260810 21:13:30.992801 26663 raft_consensus.cc:2781] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:30.992962 26663 raft_consensus.cc:686] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 1 LEADER]: Becoming Leader. State: Replica: a38c19d8d4fc4d82920addbcede9c257, State: Running, Role: LEADER
I20260810 21:13:30.993059 26663 consensus_queue.cc:237] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } }
I20260810 21:13:30.993871 25763 catalog_manager.cc:5495] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 reported cstate change: term changed from 0 to 1, leader changed from <none> to a38c19d8d4fc4d82920addbcede9c257 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "a38c19d8d4fc4d82920addbcede9c257" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38c19d8d4fc4d82920addbcede9c257" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260810 21:13:31.007843 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.005784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.27.117.65:41301)
0810 21:13:31.005838 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.005840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.005848 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:13:31.005898 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.005906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.005920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.005979 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:13:31.006077 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.006516 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.007199 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.007371 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.007423 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.007559 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:13:31.007643 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:13:31.007701 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.007735 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.007750 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 21:13:31.009524 26538 raft_consensus.cc:1261] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 1 FOLLOWER]: Refusing update from remote peer a38c19d8d4fc4d82920addbcede9c257: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:31.009581 26283 raft_consensus.cc:1261] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 1 FOLLOWER]: Refusing update from remote peer a38c19d8d4fc4d82920addbcede9c257: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:31.009868 26669 consensus_queue.cc:1035] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3663edb7aca246be93ac41c8b151ef9e" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 40841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:31.010016 26663 consensus_queue.cc:1035] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [LEADER]: Connected to new peer: Peer: permanent_uuid: "472fcac0afc041098fe095e9fef56953" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:31.012387 26678 mvcc.cc:205] Tried to move back new op lower bound from 7317079699492638720 to 7317079699428184064. Current Snapshot: MvccSnapshot[applied={T|T < 7317079699492638720}]
I20260810 21:13:31.013072 26679 mvcc.cc:205] Tried to move back new op lower bound from 7317079699492638720 to 7317079699428184064. Current Snapshot: MvccSnapshot[applied={T|T < 7317079699492638720}]
I20260810 21:13:31.014739 25763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:31.014834 25763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:31.015579 25763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0277e39c7984c61bd09e4413828d676: Sending DeleteTablet for 3 replicas of tablet 03da8abac27d4765abc37989ebdcd67b
I20260810 21:13:31.015929 26505 tablet_service.cc:1514] Processing DeleteTablet for tablet 03da8abac27d4765abc37989ebdcd67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:31 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:33358
I20260810 21:13:31.015944 26249 tablet_service.cc:1514] Processing DeleteTablet for tablet 03da8abac27d4765abc37989ebdcd67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:31 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:60932
I20260810 21:13:31.016026 26005 tablet_service.cc:1514] Processing DeleteTablet for tablet 03da8abac27d4765abc37989ebdcd67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:31 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:49324
I20260810 21:13:31.016232 26693 tablet_replica.cc:330] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257: stopping tablet replica
I20260810 21:13:31.016240 26692 tablet_replica.cc:330] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: stopping tablet replica
I20260810 21:13:31.016331 26692 raft_consensus.cc:2229] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:31.016332 26693 raft_consensus.cc:2229] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:13:31.016431 26692 raft_consensus.cc:2258] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:31.016443 26693 raft_consensus.cc:2258] T 03da8abac27d4765abc37989ebdcd67b P a38c19d8d4fc4d82920addbcede9c257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:31.016582 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 25885
I20260810 21:13:31.016973 26692 ts_tablet_manager.cc:1905] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:31.019093 26692 ts_tablet_manager.cc:1918] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:31.019168 26692 log.cc:1196] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/03da8abac27d4765abc37989ebdcd67b
I20260810 21:13:31.019517 26692 ts_tablet_manager.cc:1939] T 03da8abac27d4765abc37989ebdcd67b P 3663edb7aca246be93ac41c8b151ef9e: Deleting consensus metadata
I20260810 21:13:31.020931 26691 tablet_replica.cc:330] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: stopping tablet replica
I20260810 21:13:31.021046 26691 raft_consensus.cc:2229] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:31.021150 26691 raft_consensus.cc:2258] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:31.021490 25737 catalog_manager.cc:4841] TS 3663edb7aca246be93ac41c8b151ef9e (127.27.117.67:40841): tablet 03da8abac27d4765abc37989ebdcd67b (table test-table [id=eb7b93206e094d0bbb743f7729b29178]) successfully deleted
I20260810 21:13:31.021870 26691 ts_tablet_manager.cc:1905] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:31.023065 26691 ts_tablet_manager.cc:1918] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:31.023156 26691 log.cc:1196] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/03da8abac27d4765abc37989ebdcd67b
I20260810 21:13:31.023404 26691 ts_tablet_manager.cc:1939] T 03da8abac27d4765abc37989ebdcd67b P 472fcac0afc041098fe095e9fef56953: Deleting consensus metadata
I20260810 21:13:31.023818 25736 catalog_manager.cc:4841] TS 472fcac0afc041098fe095e9fef56953 (127.27.117.66:33255): tablet 03da8abac27d4765abc37989ebdcd67b (table test-table [id=eb7b93206e094d0bbb743f7729b29178]) successfully deleted
W20260810 21:13:31.025931 25733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:41301 (error 108)
I20260810 21:13:31.026407 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.026279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:41301 (local address 127.0.0.1:49344)
0810 21:13:31.026330 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.026374 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:41301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 21:13:31.026453 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 26138
W20260810 21:13:31.026561 25733 catalog_manager.cc:4584] TS a38c19d8d4fc4d82920addbcede9c257 (127.27.117.65:41301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03da8abac27d4765abc37989ebdcd67b: Network error: Client connection negotiation failed: client connection to 127.27.117.65:41301: connect: Connection refused (error 111)
I20260810 21:13:31.037380 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 26384
I20260810 21:13:31.048012 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 25701
2026-08-10T21:13:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T21:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:31Z Disabled control of system clock
I20260810 21:13:31.080837 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38935
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:38935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:31.162082 26701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:31.162304 26701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:31.162325 26701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:31.163399 26701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:31.163432 26701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:31.163445 26701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:31.163458 26701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:31.163470 26701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:31.164845 26701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:38935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:38935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:31.165045 26701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:31.165287 26701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:13:31.170120 26701 server_base.cc:1034] running on GCE node
W20260810 21:13:31.170092 26714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:31.170092 26717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:31.170083 26715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:31.170588 26701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:31.170907 26701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:31.172168 26701 hybrid_clock.cc:648] HybridClock initialized: now 1786396411172155 us; error 35 us; skew 500 ppm
I20260810 21:13:31.173447 26701 webserver.cc:460] Webserver started at http://127.27.117.126:41025/ using document root <none> and password file <none>
I20260810 21:13:31.173684 26701 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:31.173730 26701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:31.173954 26701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:31.175293 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "8a43c6d1a6084d08823fe4232c6cdf59"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "8ff99e2b8a9e98805c7e5552826bb3d2"
server_key_iv: "02e088c5840e4e74b2483ffaa37b1c12"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.175690 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "8a43c6d1a6084d08823fe4232c6cdf59"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "8ff99e2b8a9e98805c7e5552826bb3d2"
server_key_iv: "02e088c5840e4e74b2483ffaa37b1c12"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.177068 26701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:31.177804 26727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.177983 26701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:31.178092 26701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "8a43c6d1a6084d08823fe4232c6cdf59"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "8ff99e2b8a9e98805c7e5552826bb3d2"
server_key_iv: "02e088c5840e4e74b2483ffaa37b1c12"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.178165 26701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:31.190894 26701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:31.191226 26701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:31.191373 26701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:31.195317 26701 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:38935
I20260810 21:13:31.195364 26831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:38935 every 8 connection(s)
I20260810 21:13:31.195711 26701 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:31.196735 26834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:31.198962 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59: Bootstrap starting.
I20260810 21:13:31.199474 26834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:31.199738 26834 log.cc:824] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:31.200363 26834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59: No bootstrap required, opened a new log
I20260810 21:13:31.202008 26834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38935 } }
I20260810 21:13:31.202142 26834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:31.202163 26834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a43c6d1a6084d08823fe4232c6cdf59, State: Initialized, Role: FOLLOWER
I20260810 21:13:31.202292 26834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38935 } }
I20260810 21:13:31.202371 26834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:31.202395 26834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:31.202422 26834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:31.202899 26834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38935 } }
I20260810 21:13:31.202984 26834 leader_election.cc:302] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a43c6d1a6084d08823fe4232c6cdf59; no voters:
I20260810 21:13:31.203145 26834 leader_election.cc:288] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:31.203238 26839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:31.203421 26839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [term 1 LEADER]: Becoming Leader. State: Replica: 8a43c6d1a6084d08823fe4232c6cdf59, State: Running, Role: LEADER
I20260810 21:13:31.203444 26834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:31.203532 26839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38935 } }
I20260810 21:13:31.203933 26841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38935 } } }
I20260810 21:13:31.204010 26842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a43c6d1a6084d08823fe4232c6cdf59. Latest consensus state: current_term: 1 leader_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c6d1a6084d08823fe4232c6cdf59" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 38935 } } }
I20260810 21:13:31.204111 26841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:31.204123 26842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:31.204510 26859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:31.205070 26859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:31.205236 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 26701
I20260810 21:13:31.205341 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:31.205528 28117 external_mini_cluster.cc:1428] Setting key a5d3b401a0b4b2aa76547f78a84199f8
I20260810 21:13:31.206528 26859 catalog_manager.cc:1347] Generated new cluster ID: 8a9546acaa47418dad7b731763d0e600
I20260810 21:13:31.206583 26859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:31.211863 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.205976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.27.117.126:38935)
0810 21:13:31.206566 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.206574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.206604 (+ 30us) server_negotiation.cc:407] Connection header received
0810 21:13:31.206714 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.206721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.206785 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.206926 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:31.207152 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.208543 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.209647 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.210294 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.210421 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.211130 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:31.211134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:31.211134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:31.211341 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:31.211383 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:31.211398 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:31.211466 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.211545 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.211635 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":71,"threads_started":1}
I20260810 21:13:31.219794 26859 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:31.220469 26859 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:31.225744 26859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59: Generated new TSK 0
I20260810 21:13:31.225905 26859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:31.231343 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38935
--builtin_ntp_servers=127.27.117.84:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:31.315357 26888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:31.315600 26888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:31.315639 26888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:31.316903 26888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:31.316959 26888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:31.317003 26888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:31.318336 26888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:31.318614 26888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:31.318913 26888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:31.325086 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:31.325131 26900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:31.325343 26888 server_base.cc:1034] running on GCE node
W20260810 21:13:31.325495 26903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:31.325721 26888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:31.326011 26888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:31.327171 26888 hybrid_clock.cc:648] HybridClock initialized: now 1786396411327146 us; error 46 us; skew 500 ppm
I20260810 21:13:31.328584 26888 webserver.cc:460] Webserver started at http://127.27.117.65:41601/ using document root <none> and password file <none>
I20260810 21:13:31.328866 26888 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:31.328925 26888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:31.329068 26888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:31.330513 26888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "d8a3edd2d33046c4a6e71aab5dc5e522"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "0948230c829da9135f164ac91fffd561"
server_key_iv: "cbd5516e534890f722a9f95fcb33d6ce"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.330893 26888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "d8a3edd2d33046c4a6e71aab5dc5e522"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "0948230c829da9135f164ac91fffd561"
server_key_iv: "cbd5516e534890f722a9f95fcb33d6ce"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.332505 26888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:31.333489 26912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.333711 26888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:31.333796 26888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "d8a3edd2d33046c4a6e71aab5dc5e522"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "0948230c829da9135f164ac91fffd561"
server_key_iv: "cbd5516e534890f722a9f95fcb33d6ce"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.333851 26888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:31.351958 26888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:31.352262 26888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:31.352371 26888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:31.352576 26888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:31.353011 26888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:31.353048 26888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.353102 26888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:31.353127 26888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.359670 26888 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:42845
I20260810 21:13:31.359723 27121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:42845 every 8 connection(s)
I20260810 21:13:31.360072 26888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:13:31.365284 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.360848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:57373 (local address 127.27.117.126:38935)
0810 21:13:31.360903 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.360905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.361167 (+ 262us) server_negotiation.cc:407] Connection header received
0810 21:13:31.361671 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.361672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.361686 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.361756 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:31.362442 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.362962 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.363933 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.364126 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.364200 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.365040 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:31.365042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:31.365042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:31.365047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:31.365062 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:31.365066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:31.365116 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.365197 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.365224 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:13:31.365455 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.360764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38935 (local address 127.27.117.65:57373)
0810 21:13:31.361020 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.361063 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.361584 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.361807 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.361811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.361818 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:31.362364 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.362369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.363056 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.363057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.363838 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.363840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.364223 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.364224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.364332 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.364906 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:31.364907 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:31.364934 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:31.364956 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:31.365126 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:31.365131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:31.365133 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.365185 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.365264 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":112,"threads_started":1}
I20260810 21:13:31.366001 27122 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38935
I20260810 21:13:31.366104 27122 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:31.366258 27122 heartbeater.cc:511] Master 127.27.117.126:38935 requested a full tablet report, sending...
I20260810 21:13:31.366811 26762 ts_manager.cc:194] Registered new tserver with Master: d8a3edd2d33046c4a6e71aab5dc5e522 (127.27.117.65:42845)
I20260810 21:13:31.367040 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 26888
I20260810 21:13:31.367141 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:13:31.367321 28117 external_mini_cluster.cc:1428] Setting key 23620926a8b78339753c60e335d5ff4b
I20260810 21:13:31.367624 26762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.65:57373
I20260810 21:13:31.368572 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38935
--builtin_ntp_servers=127.27.117.84:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:31.451432 27127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:31.451628 27127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:31.451650 27127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:31.452715 27127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:31.452746 27127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:31.452770 27127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:31.454139 27127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:31.454329 27127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:31.454582 27127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:31.459501 27140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:31.459506 27134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:31.459542 27135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:31.459538 27127 server_base.cc:1034] running on GCE node
I20260810 21:13:31.459926 27127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:31.460168 27127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:31.461319 27127 hybrid_clock.cc:648] HybridClock initialized: now 1786396411461308 us; error 31 us; skew 500 ppm
I20260810 21:13:31.462513 27127 webserver.cc:460] Webserver started at http://127.27.117.66:42589/ using document root <none> and password file <none>
I20260810 21:13:31.462761 27127 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:31.462806 27127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:31.462970 27127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:31.463953 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "0fca811dc14e4d9295b65b26b7763588"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "80b134734cc672f2946d8bd42a535ddc"
server_key_iv: "4282ba99c3d4f07afc49409e6715443a"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.464298 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "0fca811dc14e4d9295b65b26b7763588"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "80b134734cc672f2946d8bd42a535ddc"
server_key_iv: "4282ba99c3d4f07afc49409e6715443a"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.465631 27127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:31.466365 27151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.466557 27127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:31.466662 27127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "0fca811dc14e4d9295b65b26b7763588"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "80b134734cc672f2946d8bd42a535ddc"
server_key_iv: "4282ba99c3d4f07afc49409e6715443a"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.466740 27127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:31.475185 27127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:31.475445 27127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:31.475605 27127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:31.475813 27127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:31.476126 27127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:31.476182 27127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.476220 27127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:31.476248 27127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.482381 27127 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:41957
I20260810 21:13:31.482437 27353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:41957 every 8 connection(s)
I20260810 21:13:31.482724 27127 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:13:31.484558 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 27127
I20260810 21:13:31.484655 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:13:31.484900 28117 external_mini_cluster.cc:1428] Setting key aa9b1e5966ec58d8be47a1fe007977f6
I20260810 21:13:31.485931 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38935
--builtin_ntp_servers=127.27.117.84:40845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 21:13:31.488694 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.483392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:39679 (local address 127.27.117.126:38935)
0810 21:13:31.483457 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.483460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.483695 (+ 235us) server_negotiation.cc:407] Connection header received
0810 21:13:31.484142 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.484143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.484152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.484207 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:31.485076 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.485787 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.487242 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.487428 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.487484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.488380 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:31.488381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:31.488382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:31.488387 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:31.488403 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:31.488408 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:31.488468 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.488536 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.488573 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 21:13:31.488831 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.483386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38935 (local address 127.27.117.66:39679)
0810 21:13:31.483581 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.483613 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.484061 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.484251 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.484254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.484261 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:31.484984 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.484991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.485908 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.485909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.487155 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.487158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.487515 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.487516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.487691 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.488244 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:31.488245 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:31.488270 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:31.488290 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:31.488474 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:31.488478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:31.488481 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.488528 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.488627 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260810 21:13:31.489274 27355 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38935
I20260810 21:13:31.489375 27355 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:31.489562 27355 heartbeater.cc:511] Master 127.27.117.126:38935 requested a full tablet report, sending...
I20260810 21:13:31.489920 26761 ts_manager.cc:194] Registered new tserver with Master: 0fca811dc14e4d9295b65b26b7763588 (127.27.117.66:41957)
I20260810 21:13:31.490283 26761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.66:39679
W20260810 21:13:31.567132 27359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:31.567309 27359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:31.567330 27359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:31.568480 27359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:31.568522 27359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:31.568547 27359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:31.569993 27359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:38935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:31.570231 27359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:31.570551 27359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:31.575475 27376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:31.575475 27373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:31.575659 27359 server_base.cc:1034] running on GCE node
W20260810 21:13:31.575713 27371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:31.575937 27359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:31.576138 27359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:31.577299 27359 hybrid_clock.cc:648] HybridClock initialized: now 1786396411577281 us; error 29 us; skew 500 ppm
I20260810 21:13:31.578428 27359 webserver.cc:460] Webserver started at http://127.27.117.67:46765/ using document root <none> and password file <none>
I20260810 21:13:31.578681 27359 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:31.578752 27359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:31.578907 27359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:31.579927 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "123833dfd8264f2b858f9bced3d6dbae"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "a791a20e53b42bed45d639273f2f25d9"
server_key_iv: "a555a5863a505ecc6534a30902baaa00"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.580334 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "123833dfd8264f2b858f9bced3d6dbae"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "a791a20e53b42bed45d639273f2f25d9"
server_key_iv: "a555a5863a505ecc6534a30902baaa00"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.581872 27359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:31.582691 27387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.582875 27359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:31.582962 27359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "123833dfd8264f2b858f9bced3d6dbae"
format_stamp: "Formatted at 2026-08-10 21:13:31 on dist-test-slave-2kcd"
server_key: "a791a20e53b42bed45d639273f2f25d9"
server_key_iv: "a555a5863a505ecc6534a30902baaa00"
server_key_version: "encryptionkey@0"
I20260810 21:13:31.583055 27359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:31.598215 27359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:31.598467 27359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:31.598610 27359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:31.598816 27359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:31.599119 27359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:31.599172 27359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.599211 27359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:31.599237 27359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:31.606081 27359 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:45037
I20260810 21:13:31.606114 27586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:45037 every 8 connection(s)
I20260810 21:13:31.606432 27359 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:31.611147 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.607122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:53761 (local address 127.27.117.126:38935)
0810 21:13:31.607163 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.607165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.607376 (+ 211us) server_negotiation.cc:407] Connection header received
0810 21:13:31.607779 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.607780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.607789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.607842 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:31.608428 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.608977 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.609863 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.610040 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.610089 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.610864 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:31.610865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:31.610865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:31.610870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:31.610887 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:31.610891 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:31.610949 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.611041 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.611072 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 21:13:31.611346 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.607080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:38935 (local address 127.27.117.67:53761)
0810 21:13:31.607255 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.607292 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.607705 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.607885 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.607888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.607896 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:31.608355 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.608359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.609057 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.609057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.609790 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.609792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.610113 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.610114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.610214 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.610699 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:31.610699 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:31.610725 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:31.610744 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:31.610951 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:31.610957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:31.610961 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.611043 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.611136 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260810 21:13:31.611862 27588 heartbeater.cc:347] Connected to a master server at 127.27.117.126:38935
I20260810 21:13:31.611980 27588 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:31.612090 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 27359
I20260810 21:13:31.612200 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:13:31.612197 27588 heartbeater.cc:511] Master 127.27.117.126:38935 requested a full tablet report, sending...
I20260810 21:13:31.612401 28117 external_mini_cluster.cc:1428] Setting key 8dbb8824799e01c76ffc130d15050ff3
I20260810 21:13:31.612576 26761 ts_manager.cc:194] Registered new tserver with Master: 123833dfd8264f2b858f9bced3d6dbae (127.27.117.67:45037)
I20260810 21:13:31.613066 26761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.67:53761
I20260810 21:13:31.613349 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:13:31.617421 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.614641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.27.117.126:38935)
0810 21:13:31.614677 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.614679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.614788 (+ 109us) server_negotiation.cc:407] Connection header received
0810 21:13:31.614863 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.614864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.614875 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.614922 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:31.615078 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.615523 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.616291 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.616477 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.616547 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.617185 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:31.617186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:31.617187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:31.617191 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:31.617202 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:31.617206 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:31.617259 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.617318 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.617338 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:13:31.618757 26761 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:13:31.620286 26761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:13:31.626191 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.622397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45037 (local address 127.0.0.1:55192)
0810 21:13:31.622783 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.622791 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.622861 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.623176 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.623177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.623179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:31.623230 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.623232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.624365 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.624366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.625244 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.625247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.625899 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.625900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.625977 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.626001 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.626066 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.626115 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":84,"threads_started":1}
I20260810 21:13:31.626395 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.622565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.27.117.66:41957)
0810 21:13:31.622772 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.622776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.622787 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:13:31.622823 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.622825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.622839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.622947 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:31.623072 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.623852 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.625188 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.625940 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.626047 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.626140 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.626176 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.626212 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":124,"threads_started":1}
I20260810 21:13:31.626415 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.622199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:41957 (local address 127.0.0.1:49800)
0810 21:13:31.622351 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.622386 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.622446 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.622950 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.622951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.622953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:31.622994 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.622996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.623942 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.623943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.625098 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.625100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.626054 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.626055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.626108 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.626111 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.626149 (+ 38us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.626178 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":60,"threads_started":1}
I20260810 21:13:31.626508 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.622496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.27.117.67:45037)
0810 21:13:31.622673 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.622677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.622918 (+ 241us) server_negotiation.cc:407] Connection header received
0810 21:13:31.622953 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.622955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.622970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.623117 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:31.623306 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.624285 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.625305 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.625790 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.625892 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.625977 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.626381 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.626424 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260810 21:13:31.626642 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.622199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:42845 (local address 127.0.0.1:54994)
0810 21:13:31.622532 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.622543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.622597 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.622864 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.622867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.622869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:31.622931 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.622933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.624259 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.624260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.625270 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.625272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.626166 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.626167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.626226 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.626230 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.626260 (+ 30us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.626574 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":5,"thread_start_us":68,"threads_started":1}
I20260810 21:13:31.626667 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.622305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.27.117.65:42845)
0810 21:13:31.622684 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.622687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.622699 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:13:31.622739 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.622741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.622757 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.622837 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:31.623036 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.624174 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.625351 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.626075 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.626416 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.626519 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.626548 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.626597 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":111,"threads_started":1}
I20260810 21:13:31.627122 27475 tablet_service.cc:1467] Processing CreateTablet for tablet 10d52a2341aa43fb894d794dfcdb70f6 (DEFAULT_TABLE table=test-table [id=b2a344cd0c304889ba7889adafc50435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:31.627307 27006 tablet_service.cc:1467] Processing CreateTablet for tablet 10d52a2341aa43fb894d794dfcdb70f6 (DEFAULT_TABLE table=test-table [id=b2a344cd0c304889ba7889adafc50435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:31.627477 27475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d52a2341aa43fb894d794dfcdb70f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:31.627629 27006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d52a2341aa43fb894d794dfcdb70f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:31.627526 27238 tablet_service.cc:1467] Processing CreateTablet for tablet 10d52a2341aa43fb894d794dfcdb70f6 (DEFAULT_TABLE table=test-table [id=b2a344cd0c304889ba7889adafc50435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:31.627812 27238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d52a2341aa43fb894d794dfcdb70f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:31.630496 27610 tablet_bootstrap.cc:492] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Bootstrap starting.
I20260810 21:13:31.630900 27612 tablet_bootstrap.cc:492] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Bootstrap starting.
I20260810 21:13:31.631191 27610 tablet_bootstrap.cc:654] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:31.631494 27612 tablet_bootstrap.cc:654] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:31.631494 27610 log.cc:824] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:31.631776 27612 log.cc:824] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:31.632128 27611 tablet_bootstrap.cc:492] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: Bootstrap starting.
I20260810 21:13:31.632180 27610 tablet_bootstrap.cc:492] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: No bootstrap required, opened a new log
I20260810 21:13:31.632268 27610 ts_tablet_manager.cc:1397] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:13:31.632491 27612 tablet_bootstrap.cc:492] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: No bootstrap required, opened a new log
I20260810 21:13:31.632545 27612 ts_tablet_manager.cc:1397] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:13:31.633484 27611 tablet_bootstrap.cc:654] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:31.633812 27611 log.cc:824] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:31.634688 27611 tablet_bootstrap.cc:492] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: No bootstrap required, opened a new log
I20260810 21:13:31.634766 27611 ts_tablet_manager.cc:1397] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 21:13:31.634788 27610 raft_consensus.cc:348] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.634960 27610 raft_consensus.cc:374] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:31.635003 27610 raft_consensus.cc:729] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fca811dc14e4d9295b65b26b7763588, State: Initialized, Role: FOLLOWER
I20260810 21:13:31.635118 27610 consensus_queue.cc:260] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.635155 27612 raft_consensus.cc:348] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.635354 27612 raft_consensus.cc:374] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:31.635403 27612 raft_consensus.cc:729] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123833dfd8264f2b858f9bced3d6dbae, State: Initialized, Role: FOLLOWER
I20260810 21:13:31.635426 27610 ts_tablet_manager.cc:1428] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:13:31.635555 27355 heartbeater.cc:503] Master 127.27.117.126:38935 was elected leader, sending a full tablet report...
I20260810 21:13:31.635548 27612 consensus_queue.cc:260] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.635823 27612 ts_tablet_manager.cc:1428] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 21:13:31.635823 27588 heartbeater.cc:503] Master 127.27.117.126:38935 was elected leader, sending a full tablet report...
I20260810 21:13:31.637205 27611 raft_consensus.cc:348] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.637323 27611 raft_consensus.cc:374] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:31.637344 27611 raft_consensus.cc:729] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a3edd2d33046c4a6e71aab5dc5e522, State: Initialized, Role: FOLLOWER
I20260810 21:13:31.637446 27611 consensus_queue.cc:260] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.637676 27611 ts_tablet_manager.cc:1428] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:31.637777 27122 heartbeater.cc:503] Master 127.27.117.126:38935 was elected leader, sending a full tablet report...
W20260810 21:13:31.733255 27356 tablet.cc:2369] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:31.856419 27621 raft_consensus.cc:482] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:31.856585 27621 raft_consensus.cc:504] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
W20260810 21:13:31.856937 27589 tablet.cc:2369] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:31.857236 27621 leader_election.cc:288] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8a3edd2d33046c4a6e71aab5dc5e522 (127.27.117.65:42845), 0fca811dc14e4d9295b65b26b7763588 (127.27.117.66:41957)
I20260810 21:13:31.859911 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.857211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:42845 (local address 127.27.117.67:39647)
0810 21:13:31.857281 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.857293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.857339 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.857492 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.857493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.857495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:31.857575 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.857577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.858317 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.858318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.859109 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.859111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.859661 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.859662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.859738 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.859741 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.859802 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.859835 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 21:13:31.859937 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.857276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:39647 (local address 127.27.117.65:42845)
0810 21:13:31.857320 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.857322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.857332 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:31.857400 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.857401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.857412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.857475 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:31.857672 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.858235 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.859199 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.859577 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.859634 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.859640 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.859814 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.859855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:13:31.860025 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.857219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:41957 (local address 127.27.117.67:55171)
0810 21:13:31.857504 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.857514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:31.857569 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:31.857713 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:31.857714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:31.857716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:31.857782 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.857784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.858433 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.858434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.859199 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:31.859201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.859750 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.859751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:31.859847 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.859851 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:31.859910 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 21:13:31.859945 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":105,"thread_start_us":102,"threads_started":1}
I20260810 21:13:31.860070 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.857284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:55171 (local address 127.27.117.66:41957)
0810 21:13:31.857330 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.857332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.857592 (+ 260us) server_negotiation.cc:407] Connection header received
0810 21:13:31.857628 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.857629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.857640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.857699 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:31.857855 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.858354 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.859291 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.859671 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.859740 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.859748 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.859942 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.859989 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:13:31.860231 27042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d52a2341aa43fb894d794dfcdb70f6" candidate_uuid: "123833dfd8264f2b858f9bced3d6dbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" is_pre_election: true
I20260810 21:13:31.860373 27042 raft_consensus.cc:2454] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123833dfd8264f2b858f9bced3d6dbae in term 0.
I20260810 21:13:31.860441 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d52a2341aa43fb894d794dfcdb70f6" candidate_uuid: "123833dfd8264f2b858f9bced3d6dbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fca811dc14e4d9295b65b26b7763588" is_pre_election: true
I20260810 21:13:31.860579 27264 raft_consensus.cc:2454] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123833dfd8264f2b858f9bced3d6dbae in term 0.
I20260810 21:13:31.860630 27388 leader_election.cc:302] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123833dfd8264f2b858f9bced3d6dbae, d8a3edd2d33046c4a6e71aab5dc5e522; no voters:
I20260810 21:13:31.860754 27621 raft_consensus.cc:2781] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 21:13:31.860761 27123 tablet.cc:2369] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:31.860817 27621 raft_consensus.cc:482] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:13:31.860841 27621 raft_consensus.cc:3037] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:31.861558 27621 raft_consensus.cc:504] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.861743 27621 leader_election.cc:288] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [CANDIDATE]: Term 1 election: Requested vote from peers d8a3edd2d33046c4a6e71aab5dc5e522 (127.27.117.65:42845), 0fca811dc14e4d9295b65b26b7763588 (127.27.117.66:41957)
I20260810 21:13:31.861873 27042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d52a2341aa43fb894d794dfcdb70f6" candidate_uuid: "123833dfd8264f2b858f9bced3d6dbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522"
I20260810 21:13:31.861894 27264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d52a2341aa43fb894d794dfcdb70f6" candidate_uuid: "123833dfd8264f2b858f9bced3d6dbae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fca811dc14e4d9295b65b26b7763588"
I20260810 21:13:31.861969 27042 raft_consensus.cc:3037] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:31.861967 27264 raft_consensus.cc:3037] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:31.862694 27264 raft_consensus.cc:2454] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123833dfd8264f2b858f9bced3d6dbae in term 1.
I20260810 21:13:31.862782 27042 raft_consensus.cc:2454] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123833dfd8264f2b858f9bced3d6dbae in term 1.
I20260810 21:13:31.862890 27391 leader_election.cc:302] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fca811dc14e4d9295b65b26b7763588, 123833dfd8264f2b858f9bced3d6dbae; no voters:
I20260810 21:13:31.863018 27621 raft_consensus.cc:2781] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:31.863219 27621 raft_consensus.cc:686] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 1 LEADER]: Becoming Leader. State: Replica: 123833dfd8264f2b858f9bced3d6dbae, State: Running, Role: LEADER
I20260810 21:13:31.863325 27621 consensus_queue.cc:237] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } }
I20260810 21:13:31.864269 26761 catalog_manager.cc:5495] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae reported cstate change: term changed from 0 to 1, leader changed from <none> to 123833dfd8264f2b858f9bced3d6dbae (127.27.117.67). New cstate: current_term: 1 leader_uuid: "123833dfd8264f2b858f9bced3d6dbae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123833dfd8264f2b858f9bced3d6dbae" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45037 } health_report { overall_health: HEALTHY } } }
I20260810 21:13:31.908030 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.905556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.27.117.67:45037)
0810 21:13:31.905636 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:31.905638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:31.905647 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:13:31.905692 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:31.905693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:31.905713 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:31.905799 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:13:31.905947 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.906433 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.907201 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:31.907416 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:31.907500 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:31.907675 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:13:31.907788 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:13:31.907886 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:31.907904 (+ 18us) server_negotiation.cc:299] Negotiation successful
0810 21:13:31.907922 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 21:13:31.909970 27264 raft_consensus.cc:1261] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 1 FOLLOWER]: Refusing update from remote peer 123833dfd8264f2b858f9bced3d6dbae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:31.910018 27042 raft_consensus.cc:1261] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 1 FOLLOWER]: Refusing update from remote peer 123833dfd8264f2b858f9bced3d6dbae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:31.910390 27624 consensus_queue.cc:1035] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fca811dc14e4d9295b65b26b7763588" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:31.910538 27621 consensus_queue.cc:1035] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8a3edd2d33046c4a6e71aab5dc5e522" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 42845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:31.912832 27632 mvcc.cc:205] Tried to move back new op lower bound from 7317079703180279808 to 7317079702992846848. Current Snapshot: MvccSnapshot[applied={T|T < 7317079703180279808}]
I20260810 21:13:31.913189 27636 mvcc.cc:205] Tried to move back new op lower bound from 7317079703180279808 to 7317079702992846848. Current Snapshot: MvccSnapshot[applied={T|T < 7317079703180279808}]
I20260810 21:13:31.915285 26762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:31.915371 26762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:31.916025 26762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a43c6d1a6084d08823fe4232c6cdf59: Sending DeleteTablet for 3 replicas of tablet 10d52a2341aa43fb894d794dfcdb70f6
I20260810 21:13:31.916381 27238 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d52a2341aa43fb894d794dfcdb70f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:31 UTC) from {username='slave'} at 127.0.0.1:49800
I20260810 21:13:31.916460 27475 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d52a2341aa43fb894d794dfcdb70f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:31 UTC) from {username='slave'} at 127.0.0.1:55192
I20260810 21:13:31.916633 27650 tablet_replica.cc:330] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: stopping tablet replica
I20260810 21:13:31.916636 27006 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d52a2341aa43fb894d794dfcdb70f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:31 UTC) from {username='slave'} at 127.0.0.1:54994
I20260810 21:13:31.916742 27650 raft_consensus.cc:2229] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:31.916872 27650 raft_consensus.cc:2258] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:31.917009 27651 tablet_replica.cc:330] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: stopping tablet replica
I20260810 21:13:31.916954 27652 tablet_replica.cc:330] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522: stopping tablet replica
I20260810 21:13:31.917197 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 26888
I20260810 21:13:31.917217 27652 raft_consensus.cc:2229] T 10d52a2341aa43fb894d794dfcdb70f6 P d8a3edd2d33046c4a6e71aab5dc5e522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:31.937062 27650 ts_tablet_manager.cc:1905] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:31.938539 27651 raft_consensus.cc:2229] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:13:31.938740 27651 raft_consensus.cc:2258] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:31.939242 27651 ts_tablet_manager.cc:1905] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:31.939361 27650 ts_tablet_manager.cc:1918] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:31.939447 27650 log.cc:1196] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/10d52a2341aa43fb894d794dfcdb70f6
I20260810 21:13:31.939744 27650 ts_tablet_manager.cc:1939] T 10d52a2341aa43fb894d794dfcdb70f6 P 0fca811dc14e4d9295b65b26b7763588: Deleting consensus metadata
I20260810 21:13:31.940243 26732 catalog_manager.cc:4841] TS 0fca811dc14e4d9295b65b26b7763588 (127.27.117.66:41957): tablet 10d52a2341aa43fb894d794dfcdb70f6 (table test-table [id=b2a344cd0c304889ba7889adafc50435]) successfully deleted
I20260810 21:13:31.940948 27651 ts_tablet_manager.cc:1918] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:31.941020 27651 log.cc:1196] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/10d52a2341aa43fb894d794dfcdb70f6
I20260810 21:13:31.941205 27651 ts_tablet_manager.cc:1939] T 10d52a2341aa43fb894d794dfcdb70f6 P 123833dfd8264f2b858f9bced3d6dbae: Deleting consensus metadata
I20260810 21:13:31.941694 26730 catalog_manager.cc:4841] TS 123833dfd8264f2b858f9bced3d6dbae (127.27.117.67:45037): tablet 10d52a2341aa43fb894d794dfcdb70f6 (table test-table [id=b2a344cd0c304889ba7889adafc50435]) successfully deleted
W20260810 21:13:31.944288 26730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:42845 (error 108)
I20260810 21:13:31.944850 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:31.944688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:42845 (local address 127.0.0.1:55000)
0810 21:13:31.944747 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:31.944811 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:42845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 21:13:31.944933 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 27127
W20260810 21:13:31.945073 26730 catalog_manager.cc:4584] TS d8a3edd2d33046c4a6e71aab5dc5e522 (127.27.117.65:42845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10d52a2341aa43fb894d794dfcdb70f6: Network error: Client connection negotiation failed: client connection to 127.27.117.65:42845: connect: Connection refused (error 111)
I20260810 21:13:31.969751 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 27359
I20260810 21:13:31.979490 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 26701
2026-08-10T21:13:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:32 dist-test-slave-2kcd krb5kdc[27657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:32 dist-test-slave-2kcd krb5kdc[27657](info): set up 2 sockets
Aug 10 21:13:32 dist-test-slave-2kcd krb5kdc[27657](info): commencing operation
krb5kdc: starting...
W20260810 21:13:33.256901 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:13:33.376945 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42765
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:33.456765 27687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:33.457012 27687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:33.457050 27687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:33.458142 27687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:33.458205 27687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:33.458233 27687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:33.458266 27687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:33.458309 27687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:33.459743 27687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:42765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:33.460005 27687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:33.460222 27687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:33.464867 27699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.464883 27706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.464943 27703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:33.465066 27687 server_base.cc:1034] running on GCE node
I20260810 21:13:33.465306 27687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:33.465566 27687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:33.466717 27687 hybrid_clock.cc:648] HybridClock initialized: now 1786396413466705 us; error 29 us; skew 500 ppm
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:33.469162 27687 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:13:33.469602 27687 webserver.cc:460] Webserver started at http://127.27.117.126:38347/ using document root <none> and password file <none>
I20260810 21:13:33.469789 27687 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:33.469827 27687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:33.469919 27687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:33.470809 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "84fa5374858845f385a558f399cee982"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "093a80d100203114ab7eac887256ca11"
server_key_iv: "077d8145781eff9bd77bcaf9ea13d44f"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.471149 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "84fa5374858845f385a558f399cee982"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "093a80d100203114ab7eac887256ca11"
server_key_iv: "077d8145781eff9bd77bcaf9ea13d44f"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.472347 27687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:13:33.473038 27722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.473206 27687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:13:33.473312 27687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "84fa5374858845f385a558f399cee982"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "093a80d100203114ab7eac887256ca11"
server_key_iv: "077d8145781eff9bd77bcaf9ea13d44f"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.473383 27687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:33.492378 27687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:33.494129 27687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:33.494285 27687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:33.498348 27687 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:42765
I20260810 21:13:33.498399 27805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:42765 every 8 connection(s)
I20260810 21:13:33.498723 27687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:33.499558 27813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:33.501511 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 27687
I20260810 21:13:33.501605 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:33.501802 28117 external_mini_cluster.cc:1428] Setting key 2310aafb2a0a1b3e815486a2587ce03b
I20260810 21:13:33.501883 27813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: Bootstrap starting.
I20260810 21:13:33.502648 27813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:33.503001 27813 log.cc:824] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:33.503715 27813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: No bootstrap required, opened a new log
I20260810 21:13:33.505558 27813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa5374858845f385a558f399cee982" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42765 } }
I20260810 21:13:33.505707 27813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:33.505756 27813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84fa5374858845f385a558f399cee982, State: Initialized, Role: FOLLOWER
I20260810 21:13:33.505885 27813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa5374858845f385a558f399cee982" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42765 } }
I20260810 21:13:33.505980 27813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:33.506022 27813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:33.506063 27813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:33.506744 27813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa5374858845f385a558f399cee982" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42765 } }
I20260810 21:13:33.506870 27813 leader_election.cc:302] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84fa5374858845f385a558f399cee982; no voters:
I20260810 21:13:33.507050 27813 leader_election.cc:288] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:33.507090 27820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:33.507225 27820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [term 1 LEADER]: Becoming Leader. State: Replica: 84fa5374858845f385a558f399cee982, State: Running, Role: LEADER
I20260810 21:13:33.507302 27813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:33.507308 27820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa5374858845f385a558f399cee982" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42765 } }
I20260810 21:13:33.507642 27821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84fa5374858845f385a558f399cee982. Latest consensus state: current_term: 1 leader_uuid: "84fa5374858845f385a558f399cee982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa5374858845f385a558f399cee982" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42765 } } }
I20260810 21:13:33.507731 27821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [sys.catalog]: This master's current role is: LEADER
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:33.509107 27823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84fa5374858845f385a558f399cee982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84fa5374858845f385a558f399cee982" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 42765 } } }
I20260810 21:13:33.509203 27823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982 [sys.catalog]: This master's current role is: LEADER
W20260810 21:13:33.509217 27848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:13:33.509267 27848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:13:33.509339 27828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:33.509521 27828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:33.510968 27828 catalog_manager.cc:1347] Generated new cluster ID: 0f6056c19b3549dda6707da94cc3c71b
I20260810 21:13:33.511018 27828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:33.512434 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:33.502248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.27.117.126:42765)
0810 21:13:33.502465 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:33.502472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:33.502495 (+ 23us) server_negotiation.cc:407] Connection header received
0810 21:13:33.502947 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:33.502954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:33.502996 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:33.503170 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:33.503825 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.504630 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.505465 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.505956 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.506084 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:33.508798 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:33.508802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:33.508803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:33.508815 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:33.510797 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:33.511261 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:33.511264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:33.511264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:33.511360 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:33.511581 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:33.511582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:33.511582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:33.511807 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:33.511968 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:33.512096 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 21:13:33.512233 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":129,"threads_started":1}
I20260810 21:13:33.525844 27828 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:33.526494 27828 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:33.539942 27828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: Generated new TSK 0
I20260810 21:13:33.540093 27828 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:13:33.573504 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42765
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:33.657035 27855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:33.657258 27855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:33.657298 27855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:33.658659 27855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:33.658736 27855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:33.658784 27855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:33.660295 27855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:33.660562 27855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:33.660820 27855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:33.665773 27864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.665810 27865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:33.665941 27855 server_base.cc:1034] running on GCE node
W20260810 21:13:33.665812 27868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:33.666191 27855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:33.666464 27855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:33.667645 27855 hybrid_clock.cc:648] HybridClock initialized: now 1786396413667632 us; error 26 us; skew 500 ppm
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:33.670230 27855 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:13:33.670696 27855 webserver.cc:460] Webserver started at http://127.27.117.65:44639/ using document root <none> and password file <none>
I20260810 21:13:33.670950 27855 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:33.671028 27855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:33.671176 27855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:33.672178 27855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "85c08235a240173431f3ce3581725e9e"
server_key_iv: "f0d4bdddb71cf33562e344ace3aa4e61"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.672531 27855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "85c08235a240173431f3ce3581725e9e"
server_key_iv: "f0d4bdddb71cf33562e344ace3aa4e61"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.673960 27855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:33.674697 27881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.674865 27855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:33.674968 27855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "85c08235a240173431f3ce3581725e9e"
server_key_iv: "f0d4bdddb71cf33562e344ace3aa4e61"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.675058 27855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:33.684845 27855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:33.687086 27855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:33.687247 27855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:33.687464 27855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:33.687762 27855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:33.687817 27855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.687856 27855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:33.687883 27855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.694052 27855 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:33281
I20260810 21:13:33.694092 28100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:33281 every 8 connection(s)
I20260810 21:13:33.694401 27855 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:13:33.698834 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 27855
I20260810 21:13:33.698953 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:13:33.699174 28117 external_mini_cluster.cc:1428] Setting key afeaa81f886a3d1e1bd9e41fab5874b4
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:33.703651 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:33.695117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:41369 (local address 127.27.117.126:42765)
0810 21:13:33.695186 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:33.695188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:33.695402 (+ 214us) server_negotiation.cc:407] Connection header received
0810 21:13:33.695863 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:33.695864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:33.695873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:33.695931 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:33.696818 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.697331 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.698252 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.698456 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.698515 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:33.700627 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:33.700628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:33.700628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:33.700637 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:33.702203 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:33.702689 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:33.702690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:33.702690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:33.702742 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:33.703059 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:33.703059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:33.703060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:33.703155 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:33.703206 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:33.703520 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 21:13:33.703586 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:13:33.703949 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:33.695088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42765 (local address 127.27.117.65:41369)
0810 21:13:33.695263 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:33.695303 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:33.695779 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:33.695979 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:33.695982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:33.696518 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:33.696701 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:33.696705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.697420 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.697420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:33.698175 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:33.698177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.698531 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.698532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:33.698633 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:33.699159 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:33.699167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:33.700534 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:33.702282 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:33.702289 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:33.702291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:33.702607 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:33.702821 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:33.702821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:33.702822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:33.702974 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:33.703223 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:33.703230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:33.703385 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 21:13:33.703500 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 21:13:33.703673 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260810 21:13:33.704454 28101 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42765
I20260810 21:13:33.704566 28101 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:33.704739 28101 heartbeater.cc:511] Master 127.27.117.126:42765 requested a full tablet report, sending...
I20260810 21:13:33.705323 27745 ts_manager.cc:194] Registered new tserver with Master: 8d2fccc7bde34b7d8fd3a44eae71bf49 (127.27.117.65:33281)
I20260810 21:13:33.705929 27745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:41369
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:13:33.725803 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42765
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:33.807039 28107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:33.807273 28107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:33.807310 28107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:33.808645 28107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:33.808709 28107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:33.808753 28107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:33.810254 28107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:33.810528 28107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:33.810758 28107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:33.815702 28132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.815686 28130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.815686 28134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:33.816076 28107 server_base.cc:1034] running on GCE node
I20260810 21:13:33.816259 28107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:33.816471 28107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:33.817654 28107 hybrid_clock.cc:648] HybridClock initialized: now 1786396413817641 us; error 25 us; skew 500 ppm
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:33.820097 28107 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:13:33.820569 28107 webserver.cc:460] Webserver started at http://127.27.117.66:43087/ using document root <none> and password file <none>
I20260810 21:13:33.820866 28107 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:33.820936 28107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:33.821081 28107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:33.822100 28107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "97c99199efaf41febd2f41e7bfe178cd"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "9b29585ef1105f5ce400932beb371d86"
server_key_iv: "a22762a3feb84739574faa8f443f2e5b"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.822520 28107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "97c99199efaf41febd2f41e7bfe178cd"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "9b29585ef1105f5ce400932beb371d86"
server_key_iv: "a22762a3feb84739574faa8f443f2e5b"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.823884 28107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:33.824725 28146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.825029 28107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:33.825157 28107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "97c99199efaf41febd2f41e7bfe178cd"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "9b29585ef1105f5ce400932beb371d86"
server_key_iv: "a22762a3feb84739574faa8f443f2e5b"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.825235 28107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:33.854110 28107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:33.856249 28107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:33.856554 28107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:33.856845 28107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:33.857196 28107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:33.857254 28107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.857290 28107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:33.857338 28107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.863628 28107 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36911
I20260810 21:13:33.863682 28369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36911 every 8 connection(s)
I20260810 21:13:33.864005 28107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:33.871035 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28107
I20260810 21:13:33.871165 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:13:33.871377 28117 external_mini_cluster.cc:1428] Setting key b1037274db3a7576ce2ab901c11d37ac
I20260810 21:13:33.873759 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:33.864762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:49613 (local address 127.27.117.126:42765)
0810 21:13:33.864835 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:33.864838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:33.865044 (+ 206us) server_negotiation.cc:407] Connection header received
0810 21:13:33.865578 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:33.865579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:33.865593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:33.865654 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:33.866609 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.867120 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.868131 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.868347 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.868427 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:33.870599 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:33.870601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:33.870601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:33.870609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:33.872196 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:33.872700 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:33.872701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:33.872702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:33.872757 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:33.873108 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:33.873110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:33.873110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:33.873229 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:33.873298 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:33.873625 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 21:13:33.873693 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 21:13:33.874050 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:33.864713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42765 (local address 127.27.117.66:49613)
0810 21:13:33.864916 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:33.864953 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:33.865469 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:33.865727 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:33.865730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:33.866307 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:33.866505 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:33.866510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.867222 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.867223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:33.868036 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:33.868037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:33.868434 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:33.868436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:33.868578 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:33.869131 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:33.869139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:33.870498 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:33.872282 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:33.872290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:33.872293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:33.872616 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:33.872863 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:33.872864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:33.872864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:33.873017 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:33.873306 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:33.873312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:33.873474 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 21:13:33.873596 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 21:13:33.873776 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260810 21:13:33.874698 28370 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42765
I20260810 21:13:33.874801 28370 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:33.874979 28370 heartbeater.cc:511] Master 127.27.117.126:42765 requested a full tablet report, sending...
I20260810 21:13:33.875381 27745 ts_manager.cc:194] Registered new tserver with Master: 97c99199efaf41febd2f41e7bfe178cd (127.27.117.66:36911)
I20260810 21:13:33.875785 27745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:49613
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:13:33.900029 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42765
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:33.981735 28381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:33.981918 28381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:33.981942 28381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:33.982967 28381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:33.983000 28381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:33.983024 28381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:33.984386 28381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:42765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:33.984606 28381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:33.984855 28381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:33.989734 28388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.989708 28390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:33.989719 28387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:33.989753 28381 server_base.cc:1034] running on GCE node
I20260810 21:13:33.990157 28381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:33.990401 28381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:33.991565 28381 hybrid_clock.cc:648] HybridClock initialized: now 1786396413991562 us; error 40 us; skew 500 ppm
Aug 10 21:13:33 dist-test-slave-2kcd krb5kdc[27657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:33.994025 28381 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:13:33.994481 28381 webserver.cc:460] Webserver started at http://127.27.117.67:38679/ using document root <none> and password file <none>
I20260810 21:13:33.994753 28381 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:33.994820 28381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:33.994961 28381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:33.995936 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "fc5f9a00991643dda04ec2b146b10fbb"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "489e4875093181be201c3b7547ae4c87"
server_key_iv: "0a59b868aef757ea71ff77cb26dd7713"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.996294 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "fc5f9a00991643dda04ec2b146b10fbb"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "489e4875093181be201c3b7547ae4c87"
server_key_iv: "0a59b868aef757ea71ff77cb26dd7713"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.997709 28381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:33.998415 28405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.998597 28381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:33.998696 28381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "fc5f9a00991643dda04ec2b146b10fbb"
format_stamp: "Formatted at 2026-08-10 21:13:33 on dist-test-slave-2kcd"
server_key: "489e4875093181be201c3b7547ae4c87"
server_key_iv: "0a59b868aef757ea71ff77cb26dd7713"
server_key_version: "encryptionkey@0"
I20260810 21:13:33.998778 28381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:34.020156 28381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:34.022107 28381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:34.022272 28381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:34.022504 28381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:34.022820 28381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:34.022893 28381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.022953 28381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:34.022984 28381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.029347 28381 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:45041
I20260810 21:13:34.029403 28628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:45041 every 8 connection(s)
I20260810 21:13:34.029731 28381 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:34.035302 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28381
I20260810 21:13:34.035408 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:13:34.035580 28117 external_mini_cluster.cc:1428] Setting key 62b4625f231bab940a36115f6d8466ad
Aug 10 21:13:34 dist-test-slave-2kcd krb5kdc[27657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:34.039491 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.030817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:34591 (local address 127.27.117.126:42765)
0810 21:13:34.030861 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.030864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.030975 (+ 111us) server_negotiation.cc:407] Connection header received
0810 21:13:34.031774 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.031775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.031786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.031839 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.032750 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.033291 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.034277 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.034455 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.034504 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.037163 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.037164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.037164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:34.037171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.038364 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:34.038725 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:34.038725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:34.038726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:34.038775 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:34.039016 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:34.039017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:34.039017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:34.039109 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.039158 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.039374 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.039431 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:13:34.039676 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.030500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:42765 (local address 127.27.117.67:34591)
0810 21:13:34.030744 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.030843 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.031650 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.031893 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.031896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.032472 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:34.032663 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.032668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.033379 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.033380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.034197 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.034200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.034530 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.034531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.034634 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.035222 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:34.035230 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:34.037072 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:34.038434 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:34.038441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:34.038442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:34.038657 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:34.038850 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:34.038851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:34.038851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:34.038951 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:34.039165 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:34.039169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:34.039272 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.039360 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.039475 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260810 21:13:34.040382 28629 heartbeater.cc:347] Connected to a master server at 127.27.117.126:42765
I20260810 21:13:34.040467 28629 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:34.040603 28629 heartbeater.cc:511] Master 127.27.117.126:42765 requested a full tablet report, sending...
I20260810 21:13:34.040971 27745 ts_manager.cc:194] Registered new tserver with Master: fc5f9a00991643dda04ec2b146b10fbb (127.27.117.67:45041)
I20260810 21:13:34.041342 27745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:34591
I20260810 21:13:34.047308 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:13:34.054468 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.048599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.27.117.126:42765)
0810 21:13:34.048674 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.048676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.048769 (+ 93us) server_negotiation.cc:407] Connection header received
0810 21:13:34.048858 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.048858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.048866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.048931 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.049430 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.049865 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.050518 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.050686 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.050736 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.051832 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.051833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.051833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:34.051839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.053246 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:34.053590 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:34.053591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:34.053591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:34.053641 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:34.053818 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:34.053819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:34.053819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:34.053936 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.054039 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.054175 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.054357 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 21:13:34.056209 27745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:13:34.057859 27745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:13:34.064263 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.060030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45041 (local address 127.0.0.1:38508)
0810 21:13:34.060119 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.060156 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.060213 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.060555 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.060558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.060559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.060634 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.060635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.061987 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.061988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.063001 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.063003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.063919 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.063920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.064006 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.064026 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.064115 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.064170 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":25,"thread_start_us":49,"threads_started":1}
I20260810 21:13:34.064448 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.060042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36911 (local address 127.0.0.1:36502)
0810 21:13:34.060168 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.060179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.060230 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.060801 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.060801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.060803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.060842 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.060843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.062058 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.062059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.063199 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.063201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.064183 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.064184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.064250 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.064255 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.064294 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.064337 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":82,"threads_started":1}
I20260810 21:13:34.064661 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.060193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.27.117.67:45041)
0810 21:13:34.060373 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.060378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.060388 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:34.060444 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.060446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.060461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.060539 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.060723 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.061885 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.063099 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.063827 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.063903 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.063958 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.064175 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.064222 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260810 21:13:34.064666 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.060415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.27.117.66:36911)
0810 21:13:34.060588 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.060592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.060604 (+ 12us) server_negotiation.cc:407] Connection header received
0810 21:13:34.060646 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.060648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.060664 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.060807 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.060906 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.061930 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.063288 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.063937 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.064419 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.064494 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.064522 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.064578 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260810 21:13:34.064698 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.060732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.27.117.65:33281)
0810 21:13:34.061040 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.061044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.061055 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:13:34.061100 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.061102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.061117 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.061199 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.061386 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.062242 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.063442 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.063942 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.064026 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.064103 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.064570 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.064621 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":136,"threads_started":1}
I20260810 21:13:34.064357 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.060225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:33281 (local address 127.0.0.1:49890)
0810 21:13:34.060340 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.060349 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.060389 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.061255 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.061256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.061257 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.061292 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.061299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.062346 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.062347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.063360 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.063362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.064139 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.064140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.064199 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.064202 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.064250 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.064287 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":82,"threads_started":1}
I20260810 21:13:34.065537 27978 tablet_service.cc:1467] Processing CreateTablet for tablet 4191bfd53ae7449b84878b8e852387b9 (DEFAULT_TABLE table=test-table [id=39bc1f15c0ca4af4b847175724012c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:34.065572 28237 tablet_service.cc:1467] Processing CreateTablet for tablet 4191bfd53ae7449b84878b8e852387b9 (DEFAULT_TABLE table=test-table [id=39bc1f15c0ca4af4b847175724012c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:34.065886 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4191bfd53ae7449b84878b8e852387b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.065907 28237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4191bfd53ae7449b84878b8e852387b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.068411 28662 tablet_bootstrap.cc:492] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: Bootstrap starting.
I20260810 21:13:34.068871 28666 tablet_bootstrap.cc:492] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Bootstrap starting.
I20260810 21:13:34.069130 28662 tablet_bootstrap.cc:654] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.069104 28499 tablet_service.cc:1467] Processing CreateTablet for tablet 4191bfd53ae7449b84878b8e852387b9 (DEFAULT_TABLE table=test-table [id=39bc1f15c0ca4af4b847175724012c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:34.069427 28499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4191bfd53ae7449b84878b8e852387b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.069530 28666 tablet_bootstrap.cc:654] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.069597 28662 log.cc:824] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.070091 28666 log.cc:824] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.070283 28662 tablet_bootstrap.cc:492] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: No bootstrap required, opened a new log
I20260810 21:13:34.070370 28662 ts_tablet_manager.cc:1397] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 21:13:34.070909 28666 tablet_bootstrap.cc:492] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: No bootstrap required, opened a new log
I20260810 21:13:34.070966 28666 ts_tablet_manager.cc:1397] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:34.071918 28662 raft_consensus.cc:348] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.072108 28662 raft_consensus.cc:374] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.072150 28662 raft_consensus.cc:729] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2fccc7bde34b7d8fd3a44eae71bf49, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.072264 28662 consensus_queue.cc:260] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.072600 28101 heartbeater.cc:503] Master 127.27.117.126:42765 was elected leader, sending a full tablet report...
I20260810 21:13:34.072715 28675 tablet_bootstrap.cc:492] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Bootstrap starting.
I20260810 21:13:34.073103 28666 raft_consensus.cc:348] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.073266 28666 raft_consensus.cc:374] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.073313 28666 raft_consensus.cc:729] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c99199efaf41febd2f41e7bfe178cd, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.073451 28675 tablet_bootstrap.cc:654] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.073431 28666 consensus_queue.cc:260] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.072583 28662 ts_tablet_manager.cc:1428] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 21:13:34.073747 28370 heartbeater.cc:503] Master 127.27.117.126:42765 was elected leader, sending a full tablet report...
I20260810 21:13:34.073736 28666 ts_tablet_manager.cc:1428] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:34.074041 28675 log.cc:824] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.074851 28675 tablet_bootstrap.cc:492] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: No bootstrap required, opened a new log
I20260810 21:13:34.074928 28675 ts_tablet_manager.cc:1397] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:34.076480 28675 raft_consensus.cc:348] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.076634 28675 raft_consensus.cc:374] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.076658 28675 raft_consensus.cc:729] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5f9a00991643dda04ec2b146b10fbb, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.076766 28675 consensus_queue.cc:260] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.077050 28675 ts_tablet_manager.cc:1428] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:34.077081 28629 heartbeater.cc:503] Master 127.27.117.126:42765 was elected leader, sending a full tablet report...
W20260810 21:13:34.114543 28371 tablet.cc:2369] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:34.167970 28670 raft_consensus.cc:482] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:34.168123 28670 raft_consensus.cc:504] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.168514 28670 leader_election.cc:288] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc5f9a00991643dda04ec2b146b10fbb (127.27.117.67:45041), 97c99199efaf41febd2f41e7bfe178cd (127.27.117.66:36911)
I20260810 21:13:34.171676 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.168921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:50419 (local address 127.27.117.66:36911)
0810 21:13:34.168976 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.168978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.169139 (+ 161us) server_negotiation.cc:407] Connection header received
0810 21:13:34.169195 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.169196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.169207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.169268 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.169470 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.169913 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.170926 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.171297 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.171380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.171390 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.171559 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.171595 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 21:13:34.171700 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.168891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:44801 (local address 127.27.117.67:45041)
0810 21:13:34.168947 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.168949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.168964 (+ 15us) server_negotiation.cc:407] Connection header received
0810 21:13:34.169040 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.169041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.169053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.169132 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.169335 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.169893 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.170885 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.171284 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.171384 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.171394 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.171585 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.171626 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 21:13:34.171718 28103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.168747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:45041 (local address 127.27.117.65:44801)
0810 21:13:34.168904 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.168919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.168974 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.169153 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.169154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.169156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.169239 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.169242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.169987 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.169988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.170779 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.170781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.171394 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.171395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.171482 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.171484 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.171559 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.171599 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":23}
I20260810 21:13:34.171729 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.168757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36911 (local address 127.27.117.65:50419)
0810 21:13:34.169053 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.169065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.169130 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.169304 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.169305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.169307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.169390 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.169392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.170041 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.170042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.170797 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.170798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.171404 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.171406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.171481 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.171485 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.171533 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.171596 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":25,"thread_start_us":138,"threads_started":1}
I20260810 21:13:34.172135 28540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4191bfd53ae7449b84878b8e852387b9" candidate_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5f9a00991643dda04ec2b146b10fbb" is_pre_election: true
I20260810 21:13:34.172169 28276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4191bfd53ae7449b84878b8e852387b9" candidate_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c99199efaf41febd2f41e7bfe178cd" is_pre_election: true
I20260810 21:13:34.172279 28540 raft_consensus.cc:2454] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2fccc7bde34b7d8fd3a44eae71bf49 in term 0.
I20260810 21:13:34.172330 28276 raft_consensus.cc:2454] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2fccc7bde34b7d8fd3a44eae71bf49 in term 0.
I20260810 21:13:34.172569 27884 leader_election.cc:302] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2fccc7bde34b7d8fd3a44eae71bf49, fc5f9a00991643dda04ec2b146b10fbb; no voters:
I20260810 21:13:34.172708 28670 raft_consensus.cc:2781] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:13:34.172771 28670 raft_consensus.cc:482] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:13:34.172837 28670 raft_consensus.cc:3037] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:34.173574 28670 raft_consensus.cc:504] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.173722 28670 leader_election.cc:288] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [CANDIDATE]: Term 1 election: Requested vote from peers fc5f9a00991643dda04ec2b146b10fbb (127.27.117.67:45041), 97c99199efaf41febd2f41e7bfe178cd (127.27.117.66:36911)
I20260810 21:13:34.173873 28540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4191bfd53ae7449b84878b8e852387b9" candidate_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc5f9a00991643dda04ec2b146b10fbb"
I20260810 21:13:34.173882 28276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4191bfd53ae7449b84878b8e852387b9" candidate_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c99199efaf41febd2f41e7bfe178cd"
I20260810 21:13:34.173969 28540 raft_consensus.cc:3037] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:34.173966 28276 raft_consensus.cc:3037] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:34.174742 28276 raft_consensus.cc:2454] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2fccc7bde34b7d8fd3a44eae71bf49 in term 1.
I20260810 21:13:34.174789 28540 raft_consensus.cc:2454] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2fccc7bde34b7d8fd3a44eae71bf49 in term 1.
I20260810 21:13:34.174955 27884 leader_election.cc:302] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2fccc7bde34b7d8fd3a44eae71bf49, fc5f9a00991643dda04ec2b146b10fbb; no voters:
I20260810 21:13:34.175046 28670 raft_consensus.cc:2781] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:34.175209 28670 raft_consensus.cc:686] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [term 1 LEADER]: Becoming Leader. State: Replica: 8d2fccc7bde34b7d8fd3a44eae71bf49, State: Running, Role: LEADER
I20260810 21:13:34.175320 28670 consensus_queue.cc:237] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } }
I20260810 21:13:34.176213 27745 catalog_manager.cc:5495] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d2fccc7bde34b7d8fd3a44eae71bf49 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d2fccc7bde34b7d8fd3a44eae71bf49" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 33281 } health_report { overall_health: HEALTHY } } }
W20260810 21:13:34.195211 28102 tablet.cc:2369] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:34.209395 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.206858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.27.117.65:33281)
0810 21:13:34.206916 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.206919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.206933 (+ 14us) server_negotiation.cc:407] Connection header received
0810 21:13:34.207027 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.207028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.207046 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.207111 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:13:34.207222 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.207725 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.208573 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.208757 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.208840 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.208977 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:13:34.209069 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:13:34.209202 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.209327 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.209338 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 21:13:34.211256 28540 raft_consensus.cc:1261] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 1 FOLLOWER]: Refusing update from remote peer 8d2fccc7bde34b7d8fd3a44eae71bf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:34.211594 28276 raft_consensus.cc:1261] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 1 FOLLOWER]: Refusing update from remote peer 8d2fccc7bde34b7d8fd3a44eae71bf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:34.211596 28670 consensus_queue.cc:1035] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc5f9a00991643dda04ec2b146b10fbb" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:34.211946 28670 consensus_queue.cc:1035] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c99199efaf41febd2f41e7bfe178cd" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:34.214159 28692 mvcc.cc:205] Tried to move back new op lower bound from 7317079712606396416 to 7317079712462725120. Current Snapshot: MvccSnapshot[applied={T|T < 7317079712606396416}]
I20260810 21:13:34.215008 28696 mvcc.cc:205] Tried to move back new op lower bound from 7317079712606396416 to 7317079712462725120. Current Snapshot: MvccSnapshot[applied={T|T < 7317079712606396416}]
I20260810 21:13:34.217407 27745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:34.217494 27745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:34.218281 27745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84fa5374858845f385a558f399cee982: Sending DeleteTablet for 3 replicas of tablet 4191bfd53ae7449b84878b8e852387b9
I20260810 21:13:34.218631 28237 tablet_service.cc:1514] Processing DeleteTablet for tablet 4191bfd53ae7449b84878b8e852387b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:36502
I20260810 21:13:34.218679 28499 tablet_service.cc:1514] Processing DeleteTablet for tablet 4191bfd53ae7449b84878b8e852387b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:38508
I20260810 21:13:34.218885 27978 tablet_service.cc:1514] Processing DeleteTablet for tablet 4191bfd53ae7449b84878b8e852387b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:34 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:49890
I20260810 21:13:34.219113 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 27855
I20260810 21:13:34.219115 28714 tablet_replica.cc:330] T 4191bfd53ae7449b84878b8e852387b9 P 8d2fccc7bde34b7d8fd3a44eae71bf49: stopping tablet replica
I20260810 21:13:34.219128 28713 tablet_replica.cc:330] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: stopping tablet replica
I20260810 21:13:34.219676 28713 raft_consensus.cc:2229] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:34.219827 28713 raft_consensus.cc:2258] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:34.220027 28710 tablet_replica.cc:330] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: stopping tablet replica
I20260810 21:13:34.220142 28710 raft_consensus.cc:2229] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:34.220255 28710 raft_consensus.cc:2258] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:34.222642 28713 ts_tablet_manager.cc:1905] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:34.224504 28713 ts_tablet_manager.cc:1918] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:34.224581 28713 log.cc:1196] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/4191bfd53ae7449b84878b8e852387b9
I20260810 21:13:34.224897 28713 ts_tablet_manager.cc:1939] T 4191bfd53ae7449b84878b8e852387b9 P fc5f9a00991643dda04ec2b146b10fbb: Deleting consensus metadata
I20260810 21:13:34.225651 27726 catalog_manager.cc:4841] TS fc5f9a00991643dda04ec2b146b10fbb (127.27.117.67:45041): tablet 4191bfd53ae7449b84878b8e852387b9 (table test-table [id=39bc1f15c0ca4af4b847175724012c59]) successfully deleted
I20260810 21:13:34.227790 28710 ts_tablet_manager.cc:1905] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:34.229894 28710 ts_tablet_manager.cc:1918] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:34.229977 28710 log.cc:1196] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/4191bfd53ae7449b84878b8e852387b9
I20260810 21:13:34.230242 28710 ts_tablet_manager.cc:1939] T 4191bfd53ae7449b84878b8e852387b9 P 97c99199efaf41febd2f41e7bfe178cd: Deleting consensus metadata
I20260810 21:13:34.230690 27727 catalog_manager.cc:4841] TS 97c99199efaf41febd2f41e7bfe178cd (127.27.117.66:36911): tablet 4191bfd53ae7449b84878b8e852387b9 (table test-table [id=39bc1f15c0ca4af4b847175724012c59]) successfully deleted
W20260810 21:13:34.232744 27726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:33281 (error 108)
I20260810 21:13:34.233269 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.233155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:33281 (local address 127.0.0.1:49904)
0810 21:13:34.233206 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.233241 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 21:13:34.233417 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28107
W20260810 21:13:34.233438 27726 catalog_manager.cc:4584] TS 8d2fccc7bde34b7d8fd3a44eae71bf49 (127.27.117.65:33281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4191bfd53ae7449b84878b8e852387b9: Network error: Client connection negotiation failed: client connection to 127.27.117.65:33281: connect: Connection refused (error 111)
I20260810 21:13:34.242920 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28381
I20260810 21:13:34.263123 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 27687
2026-08-10T21:13:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2290 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T21:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:34Z Disabled control of system clock
I20260810 21:13:34.303472 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35315
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:34.388118 28723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:34.388365 28723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:34.388386 28723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:34.389647 28723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:34.389693 28723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:34.389705 28723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:34.389719 28723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:34.389730 28723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:34.391121 28723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:34.391386 28723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:34.391698 28723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:34.396423 28732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:34.396431 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:34.396705 28723 server_base.cc:1034] running on GCE node
W20260810 21:13:34.396445 28736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:34.397063 28723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:34.397403 28723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:34.398597 28723 hybrid_clock.cc:648] HybridClock initialized: now 1786396414398587 us; error 46 us; skew 500 ppm
I20260810 21:13:34.399938 28723 webserver.cc:460] Webserver started at http://127.27.117.126:42805/ using document root <none> and password file <none>
I20260810 21:13:34.400198 28723 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:34.400271 28723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:34.400430 28723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:34.401510 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "0ce6db46ca62c74d8690e5764ca9cee0"
server_key_iv: "f3d45ce56ba43b2f7f51b09c9e55421c"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.401859 28723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "0ce6db46ca62c74d8690e5764ca9cee0"
server_key_iv: "f3d45ce56ba43b2f7f51b09c9e55421c"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.403244 28723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:13:34.403970 28748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.404150 28723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:34.404258 28723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "0ce6db46ca62c74d8690e5764ca9cee0"
server_key_iv: "f3d45ce56ba43b2f7f51b09c9e55421c"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.404331 28723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:34.414301 28723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:34.414592 28723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:34.414737 28723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:34.418732 28723 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35315
I20260810 21:13:34.418794 28853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35315 every 8 connection(s)
I20260810 21:13:34.419085 28723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:34.420028 28856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.422158 28856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a: Bootstrap starting.
I20260810 21:13:34.422751 28856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.423055 28856 log.cc:824] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.423687 28856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a: No bootstrap required, opened a new log
I20260810 21:13:34.425269 28856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35315 } }
I20260810 21:13:34.425405 28856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.425453 28856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f5269c9b4f4b2aa2dd13ade8846e9a, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.425597 28856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35315 } }
I20260810 21:13:34.425685 28856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:34.425740 28856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:34.425789 28856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:34.426265 28856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35315 } }
I20260810 21:13:34.426374 28856 leader_election.cc:302] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35f5269c9b4f4b2aa2dd13ade8846e9a; no voters:
I20260810 21:13:34.426563 28856 leader_election.cc:288] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:34.426614 28862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:34.426787 28862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [term 1 LEADER]: Becoming Leader. State: Replica: 35f5269c9b4f4b2aa2dd13ade8846e9a, State: Running, Role: LEADER
I20260810 21:13:34.426817 28856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:34.426882 28862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35315 } }
I20260810 21:13:34.427402 28863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35315 } } }
I20260810 21:13:34.427500 28863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:34.427814 28865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35f5269c9b4f4b2aa2dd13ade8846e9a. Latest consensus state: current_term: 1 leader_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35f5269c9b4f4b2aa2dd13ade8846e9a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35315 } } }
I20260810 21:13:34.427893 28865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:34.427920 28875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:34.428442 28875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:34.429119 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28723
I20260810 21:13:34.429219 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:34.429406 28117 external_mini_cluster.cc:1428] Setting key 26ccf16ce048ed67acbacf5c6683e4ca
I20260810 21:13:34.431915 28875 catalog_manager.cc:1347] Generated new cluster ID: 9a81480f8c5d486b9df42ed4084470e1
I20260810 21:13:34.431974 28875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:34.435021 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.429937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.27.117.126:35315)
0810 21:13:34.430151 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.430159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.430190 (+ 31us) server_negotiation.cc:407] Connection header received
0810 21:13:34.430297 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.430303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.430447 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.430611 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.430811 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.431760 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.432859 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.433486 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.433615 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.434295 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.434299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.434299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:34.434520 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.434561 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:34.434576 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.434647 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.434723 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.434799 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260810 21:13:34.439492 28875 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:34.439954 28875 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:34.446350 28875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a: Generated new TSK 0
I20260810 21:13:34.446533 28875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:34.452383 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35315
--builtin_ntp_servers=127.27.117.84:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:34.533453 28904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:34.533668 28904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:34.533704 28904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:34.534782 28904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:34.534844 28904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:34.534883 28904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:34.536149 28904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:34.536427 28904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:34.536713 28904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:34.542299 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:34.542335 28916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:34.542287 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:34.542626 28904 server_base.cc:1034] running on GCE node
I20260810 21:13:34.542855 28904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:34.543098 28904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:34.544261 28904 hybrid_clock.cc:648] HybridClock initialized: now 1786396414544248 us; error 31 us; skew 500 ppm
I20260810 21:13:34.545545 28904 webserver.cc:460] Webserver started at http://127.27.117.65:45073/ using document root <none> and password file <none>
I20260810 21:13:34.545791 28904 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:34.545858 28904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:34.546018 28904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:34.547030 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "50e326d8765d449db0afbeddd0497f4a"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "cc4cfe69dbe2c6061f9ec564b546e730"
server_key_iv: "8f4d4d7afc70399655cc0ce0a844bbec"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.547473 28904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "50e326d8765d449db0afbeddd0497f4a"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "cc4cfe69dbe2c6061f9ec564b546e730"
server_key_iv: "8f4d4d7afc70399655cc0ce0a844bbec"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.548869 28904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:34.549655 28927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.549813 28904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:34.549904 28904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "50e326d8765d449db0afbeddd0497f4a"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "cc4cfe69dbe2c6061f9ec564b546e730"
server_key_iv: "8f4d4d7afc70399655cc0ce0a844bbec"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.549973 28904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:34.562378 28904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:34.562677 28904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:34.562826 28904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:34.563047 28904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:34.563388 28904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:34.563439 28904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.563462 28904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:34.563515 28904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.569464 28904 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:39475
I20260810 21:13:34.569542 29134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:39475 every 8 connection(s)
I20260810 21:13:34.569867 28904 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:13:34.574798 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.570646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:50747 (local address 127.27.117.126:35315)
0810 21:13:34.570716 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.570718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.570979 (+ 261us) server_negotiation.cc:407] Connection header received
0810 21:13:34.571446 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.571448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.571458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.571515 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.572150 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.572633 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.573611 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.573799 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.573855 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.574570 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.574572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.574572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:34.574576 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.574588 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:34.574592 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.574636 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.574719 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.574742 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 21:13:34.574980 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.570646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35315 (local address 127.27.117.65:50747)
0810 21:13:34.570842 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.570880 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.571363 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.571584 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.571587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.571595 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:34.572071 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.572076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.572719 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.572720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.573526 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.573528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.573876 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.573877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.573987 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.574445 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:34.574445 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:34.574470 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:34.574490 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:34.574649 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:34.574653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:34.574655 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.574705 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.574776 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260810 21:13:34.575476 29137 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35315
I20260810 21:13:34.575587 29137 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:34.575767 29137 heartbeater.cc:511] Master 127.27.117.126:35315 requested a full tablet report, sending...
I20260810 21:13:34.576237 28784 ts_manager.cc:194] Registered new tserver with Master: 50e326d8765d449db0afbeddd0497f4a (127.27.117.65:39475)
I20260810 21:13:34.576890 28784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.65:50747
I20260810 21:13:34.578275 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 28904
I20260810 21:13:34.578357 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:13:34.578554 28117 external_mini_cluster.cc:1428] Setting key e666d443f1c8ec2c35b4ef4e9f6ccd1a
I20260810 21:13:34.579442 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35315
--builtin_ntp_servers=127.27.117.84:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:34.663615 29146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:34.663805 29146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:34.663826 29146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:34.665009 29146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:34.665050 29146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:34.665073 29146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:34.666424 29146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:34.666680 29146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:34.666991 29146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:34.672021 29157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:34.672031 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:34.672128 29146 server_base.cc:1034] running on GCE node
W20260810 21:13:34.672031 29163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:34.672482 29146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:34.672726 29146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:34.673914 29146 hybrid_clock.cc:648] HybridClock initialized: now 1786396414673899 us; error 33 us; skew 500 ppm
I20260810 21:13:34.675173 29146 webserver.cc:460] Webserver started at http://127.27.117.66:45025/ using document root <none> and password file <none>
I20260810 21:13:34.675444 29146 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:34.675514 29146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:34.675663 29146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:34.676719 29146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "e24a217f409e45ccba042a07c1e186de"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "031abfb6f6abef620762cfc349978a06"
server_key_iv: "6205392fb661bd7a69a2f96148de57b1"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.677130 29146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "e24a217f409e45ccba042a07c1e186de"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "031abfb6f6abef620762cfc349978a06"
server_key_iv: "6205392fb661bd7a69a2f96148de57b1"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.678565 29146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:34.679392 29175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.679719 29146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:34.679859 29146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "e24a217f409e45ccba042a07c1e186de"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "031abfb6f6abef620762cfc349978a06"
server_key_iv: "6205392fb661bd7a69a2f96148de57b1"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.679981 29146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:34.687208 29146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:34.687496 29146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:34.687645 29146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:34.687857 29146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:34.688180 29146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:34.688236 29146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.688277 29146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:34.688304 29146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.694348 29146 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:36239
I20260810 21:13:34.694403 29383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:36239 every 8 connection(s)
I20260810 21:13:34.694713 29146 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:13:34.699666 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.695431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:49999 (local address 127.27.117.126:35315)
0810 21:13:34.695505 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.695507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.695720 (+ 213us) server_negotiation.cc:407] Connection header received
0810 21:13:34.696193 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.696194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.696203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.696264 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.697004 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.697501 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.698445 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.698632 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.698685 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.699427 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.699428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.699429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:34.699433 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.699446 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:34.699451 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.699495 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.699584 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.699609 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 21:13:34.699880 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.695397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35315 (local address 127.27.117.66:49999)
0810 21:13:34.695587 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.695627 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.696095 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.696329 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.696332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.696339 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:34.696920 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.696925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.697580 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.697580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.698358 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.698360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.698708 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.698708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.698812 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.699290 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:34.699290 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:34.699316 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:34.699338 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:34.699508 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:34.699513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:34.699515 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.699567 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.699644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260810 21:13:34.700364 29384 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35315
I20260810 21:13:34.700467 29384 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:34.700652 29384 heartbeater.cc:511] Master 127.27.117.126:35315 requested a full tablet report, sending...
I20260810 21:13:34.701030 28784 ts_manager.cc:194] Registered new tserver with Master: e24a217f409e45ccba042a07c1e186de (127.27.117.66:36239)
I20260810 21:13:34.701392 28784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.66:49999
I20260810 21:13:34.704756 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29146
I20260810 21:13:34.704900 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:13:34.705096 28117 external_mini_cluster.cc:1428] Setting key 2930959cdc81c5482d48e5e963bda02c
I20260810 21:13:34.705927 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35315
--builtin_ntp_servers=127.27.117.84:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:34.787783 29389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:34.787977 29389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:34.787999 29389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:34.789157 29389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:34.789196 29389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:34.789220 29389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:34.790575 29389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:35315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:34.790817 29389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:34.791126 29389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:13:34.796029 29389 server_base.cc:1034] running on GCE node
W20260810 21:13:34.795975 29405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:34.796005 29400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:34.795976 29397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:34.796404 29389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:34.796671 29389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:34.797834 29389 hybrid_clock.cc:648] HybridClock initialized: now 1786396414797830 us; error 36 us; skew 500 ppm
I20260810 21:13:34.799022 29389 webserver.cc:460] Webserver started at http://127.27.117.67:46127/ using document root <none> and password file <none>
I20260810 21:13:34.799299 29389 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:34.799377 29389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:34.799513 29389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:34.800527 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "bb68d613fae64494b5cecd06e673d6b4"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "75e9a576a2284554ad79b80ea7bfbad0"
server_key_iv: "131eff00b77324dd3296add03367b6ce"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.800913 29389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "bb68d613fae64494b5cecd06e673d6b4"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "75e9a576a2284554ad79b80ea7bfbad0"
server_key_iv: "131eff00b77324dd3296add03367b6ce"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.802212 29389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:34.802975 29418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.803158 29389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:13:34.803258 29389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "bb68d613fae64494b5cecd06e673d6b4"
format_stamp: "Formatted at 2026-08-10 21:13:34 on dist-test-slave-2kcd"
server_key: "75e9a576a2284554ad79b80ea7bfbad0"
server_key_iv: "131eff00b77324dd3296add03367b6ce"
server_key_version: "encryptionkey@0"
I20260810 21:13:34.803354 29389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:34.843505 29389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:34.843812 29389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:34.843962 29389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:34.844185 29389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:34.844522 29389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:34.844576 29389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.844610 29389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:34.844637 29389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:34.850360 29389 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:42605
I20260810 21:13:34.850399 29621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:42605 every 8 connection(s)
I20260810 21:13:34.850749 29389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:34.851869 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29389
I20260810 21:13:34.851994 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:13:34.852152 28117 external_mini_cluster.cc:1428] Setting key 5fc38f5c88026f7e875392248d9590fa
I20260810 21:13:34.856003 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.851493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:49575 (local address 127.27.117.126:35315)
0810 21:13:34.851556 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.851558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.851806 (+ 248us) server_negotiation.cc:407] Connection header received
0810 21:13:34.852419 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.852421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.852433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.852499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.853396 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.853885 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.854787 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.854988 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.855041 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.855782 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.855783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.855784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:34.855788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.855800 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:34.855804 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.855868 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.855924 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.855948 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:13:34.856253 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.851488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:35315 (local address 127.27.117.67:49575)
0810 21:13:34.851670 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.851711 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.852291 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.852575 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.852580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.852589 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:34.853311 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.853318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.853965 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.853966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.854714 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.854715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.855082 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.855083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.855186 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.855657 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:34.855658 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:34.855685 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:34.855706 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:34.855871 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:34.855876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:34.855878 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.855927 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.856000 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260810 21:13:34.856797 29622 heartbeater.cc:347] Connected to a master server at 127.27.117.126:35315
I20260810 21:13:34.856900 29622 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:34.857096 29622 heartbeater.cc:511] Master 127.27.117.126:35315 requested a full tablet report, sending...
I20260810 21:13:34.857477 28784 ts_manager.cc:194] Registered new tserver with Master: bb68d613fae64494b5cecd06e673d6b4 (127.27.117.67:42605)
I20260810 21:13:34.857800 28784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.67:49575
I20260810 21:13:34.864126 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:13:34.868283 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.865594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.27.117.126:35315)
0810 21:13:34.865695 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.865697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.865729 (+ 32us) server_negotiation.cc:407] Connection header received
0810 21:13:34.865793 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.865793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.865804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.865851 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:34.866029 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.866475 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.867169 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.867339 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.867385 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.867998 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:34.867999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:34.867999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:34.868003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:34.868015 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:34.868018 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:34.868092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.868162 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.868188 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 21:13:34.869789 28784 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:13:34.871511 28784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:13:34.877767 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.873821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42605 (local address 127.0.0.1:42634)
0810 21:13:34.874049 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.874060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.874109 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.874487 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.874488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.874490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.874525 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.874526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.875822 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.875823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.876702 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.876704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.877480 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.877480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.877557 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.877571 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.877651 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.877697 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":62,"thread_start_us":92,"threads_started":1}
I20260810 21:13:34.877782 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.874017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.27.117.67:42605)
0810 21:13:34.874205 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.874211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.874221 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:34.874260 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.874262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.874277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.874368 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.874608 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.875717 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.876813 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.877400 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.877473 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.877556 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.877666 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.877700 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260810 21:13:34.878366 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.873631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:39475 (local address 127.0.0.1:57010)
0810 21:13:34.873785 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.873821 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.873882 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.874745 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.874746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.874748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.874794 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.874795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.875728 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.875729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.877081 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.877084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.878137 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.878139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.878202 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.878205 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.878259 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.878300 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":72,"thread_start_us":56,"threads_started":1}
I20260810 21:13:34.878582 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.873631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36239 (local address 127.0.0.1:54766)
0810 21:13:34.873985 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:34.873996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:34.874057 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:34.874397 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:34.874400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:34.874403 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:34.874467 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.874469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.875814 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.875814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.877289 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:34.877291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.878359 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.878360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:34.878422 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.878426 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:34.878477 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:13:34.878515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":75,"thread_start_us":86,"threads_started":1}
I20260810 21:13:34.878607 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.873725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.27.117.65:39475)
0810 21:13:34.874506 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.874511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.874521 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:34.874560 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.874562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.874578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.874664 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.874890 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.875645 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.877181 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.877940 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.878009 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.878095 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.878485 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.878532 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":107,"threads_started":1}
I20260810 21:13:34.878698 29504 tablet_service.cc:1467] Processing CreateTablet for tablet 8df2d6ca9e78454796a435ec32643cbc (DEFAULT_TABLE table=test-table [id=5fa373467bff4419b7a66c3b84c9b921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:34.878827 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:34.873818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.27.117.66:36239)
0810 21:13:34.874206 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:34.874211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:34.874221 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:13:34.874260 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:34.874262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:34.874278 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:34.874362 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:34.874563 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.875752 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.877375 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:34.878167 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:34.878235 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:34.878320 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:34.878694 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 21:13:34.878753 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":86,"threads_started":1}
I20260810 21:13:34.879109 29504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df2d6ca9e78454796a435ec32643cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.879251 29017 tablet_service.cc:1467] Processing CreateTablet for tablet 8df2d6ca9e78454796a435ec32643cbc (DEFAULT_TABLE table=test-table [id=5fa373467bff4419b7a66c3b84c9b921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:34.879463 29259 tablet_service.cc:1467] Processing CreateTablet for tablet 8df2d6ca9e78454796a435ec32643cbc (DEFAULT_TABLE table=test-table [id=5fa373467bff4419b7a66c3b84c9b921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:34.879611 29017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df2d6ca9e78454796a435ec32643cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.879770 29259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8df2d6ca9e78454796a435ec32643cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:34.882023 29649 tablet_bootstrap.cc:492] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Bootstrap starting.
I20260810 21:13:34.882678 29649 tablet_bootstrap.cc:654] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.883114 29649 log.cc:824] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.883155 29654 tablet_bootstrap.cc:492] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: Bootstrap starting.
I20260810 21:13:34.883701 29654 tablet_bootstrap.cc:654] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.883842 29649 tablet_bootstrap.cc:492] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: No bootstrap required, opened a new log
I20260810 21:13:34.883955 29649 ts_tablet_manager.cc:1397] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:13:34.884033 29654 log.cc:824] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.884387 29652 tablet_bootstrap.cc:492] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Bootstrap starting.
I20260810 21:13:34.885119 29652 tablet_bootstrap.cc:654] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:34.885406 29652 log.cc:824] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:34.885882 29654 tablet_bootstrap.cc:492] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: No bootstrap required, opened a new log
I20260810 21:13:34.885962 29654 ts_tablet_manager.cc:1397] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 21:13:34.886183 29652 tablet_bootstrap.cc:492] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: No bootstrap required, opened a new log
I20260810 21:13:34.886250 29652 ts_tablet_manager.cc:1397] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:13:34.886546 29649 raft_consensus.cc:348] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:34.886705 29649 raft_consensus.cc:374] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.886749 29649 raft_consensus.cc:729] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb68d613fae64494b5cecd06e673d6b4, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.886899 29649 consensus_queue.cc:260] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:34.887190 29649 ts_tablet_manager.cc:1428] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260810 21:13:34.887212 29622 heartbeater.cc:503] Master 127.27.117.126:35315 was elected leader, sending a full tablet report...
I20260810 21:13:34.888581 29654 raft_consensus.cc:348] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:34.888747 29654 raft_consensus.cc:374] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.888808 29654 raft_consensus.cc:729] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e326d8765d449db0afbeddd0497f4a, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.888710 29652 raft_consensus.cc:348] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:34.888890 29652 raft_consensus.cc:374] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:34.888942 29652 raft_consensus.cc:729] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24a217f409e45ccba042a07c1e186de, State: Initialized, Role: FOLLOWER
I20260810 21:13:34.888971 29654 consensus_queue.cc:260] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:34.889048 29652 consensus_queue.cc:260] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:34.889245 29654 ts_tablet_manager.cc:1428] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:34.889302 29652 ts_tablet_manager.cc:1428] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:34.889331 29137 heartbeater.cc:503] Master 127.27.117.126:35315 was elected leader, sending a full tablet report...
I20260810 21:13:34.889426 29384 heartbeater.cc:503] Master 127.27.117.126:35315 was elected leader, sending a full tablet report...
W20260810 21:13:34.945271 29387 tablet.cc:2369] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:35.031476 29662 raft_consensus.cc:482] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:35.031598 29662 raft_consensus.cc:504] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:35.032055 29662 leader_election.cc:288] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e24a217f409e45ccba042a07c1e186de (127.27.117.66:36239), bb68d613fae64494b5cecd06e673d6b4 (127.27.117.67:42605)
I20260810 21:13:35.034963 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:35.032388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:39789 (local address 127.27.117.66:36239)
0810 21:13:35.032443 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:35.032444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:35.032451 (+ 7us) server_negotiation.cc:407] Connection header received
0810 21:13:35.032475 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:35.032476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:35.032486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:35.032545 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:35.032720 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.033238 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.034232 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.034599 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.034683 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:35.034692 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:35.034846 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 21:13:35.034884 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:13:35.034977 29648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:35.032288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:56649 (local address 127.27.117.67:42605)
0810 21:13:35.032341 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:35.032344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:35.032457 (+ 113us) server_negotiation.cc:407] Connection header received
0810 21:13:35.032531 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:35.032532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:35.032541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:35.032597 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:35.032769 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.033261 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.034247 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.034610 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.034693 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:35.034701 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:35.034860 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 21:13:35.034907 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 21:13:35.034999 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:35.032228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:42605 (local address 127.27.117.65:56649)
0810 21:13:35.032391 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:35.032411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:35.032473 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:35.032605 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:35.032606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:35.032607 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:35.032681 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:35.032683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.033346 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.033347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:35.034144 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:35.034146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.034693 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.034694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:35.034778 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:35.034788 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 21:13:35.034855 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 21:13:35.034901 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":14,"thread_start_us":92,"threads_started":1}
I20260810 21:13:35.034998 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:35.032223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:36239 (local address 127.27.117.65:39789)
0810 21:13:35.032268 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:35.032281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:35.032334 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:35.032562 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:35.032563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:35.032565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:35.032648 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:35.032650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.033323 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.033324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:35.034125 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:35.034127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.034681 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.034681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:35.034751 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:35.034755 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:35.034845 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 21:13:35.034893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":22}
I20260810 21:13:35.035562 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df2d6ca9e78454796a435ec32643cbc" candidate_uuid: "50e326d8765d449db0afbeddd0497f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24a217f409e45ccba042a07c1e186de" is_pre_election: true
I20260810 21:13:35.035591 29536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df2d6ca9e78454796a435ec32643cbc" candidate_uuid: "50e326d8765d449db0afbeddd0497f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb68d613fae64494b5cecd06e673d6b4" is_pre_election: true
I20260810 21:13:35.035714 29294 raft_consensus.cc:2454] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e326d8765d449db0afbeddd0497f4a in term 0.
I20260810 21:13:35.035746 29536 raft_consensus.cc:2454] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e326d8765d449db0afbeddd0497f4a in term 0.
I20260810 21:13:35.035946 28934 leader_election.cc:302] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e326d8765d449db0afbeddd0497f4a, e24a217f409e45ccba042a07c1e186de; no voters:
I20260810 21:13:35.036067 29662 raft_consensus.cc:2781] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:13:35.036123 29662 raft_consensus.cc:482] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:13:35.036145 29662 raft_consensus.cc:3037] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:35.036906 29662 raft_consensus.cc:504] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:35.037098 29662 leader_election.cc:288] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [CANDIDATE]: Term 1 election: Requested vote from peers e24a217f409e45ccba042a07c1e186de (127.27.117.66:36239), bb68d613fae64494b5cecd06e673d6b4 (127.27.117.67:42605)
I20260810 21:13:35.037182 29294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df2d6ca9e78454796a435ec32643cbc" candidate_uuid: "50e326d8765d449db0afbeddd0497f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24a217f409e45ccba042a07c1e186de"
I20260810 21:13:35.037240 29536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8df2d6ca9e78454796a435ec32643cbc" candidate_uuid: "50e326d8765d449db0afbeddd0497f4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb68d613fae64494b5cecd06e673d6b4"
I20260810 21:13:35.037264 29294 raft_consensus.cc:3037] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:35.037297 29536 raft_consensus.cc:3037] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:35.037968 29536 raft_consensus.cc:2454] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e326d8765d449db0afbeddd0497f4a in term 1.
I20260810 21:13:35.037978 29294 raft_consensus.cc:2454] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e326d8765d449db0afbeddd0497f4a in term 1.
I20260810 21:13:35.038151 28934 leader_election.cc:302] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e326d8765d449db0afbeddd0497f4a, e24a217f409e45ccba042a07c1e186de; no voters:
I20260810 21:13:35.038233 29662 raft_consensus.cc:2781] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:35.038391 29662 raft_consensus.cc:686] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 1 LEADER]: Becoming Leader. State: Replica: 50e326d8765d449db0afbeddd0497f4a, State: Running, Role: LEADER
I20260810 21:13:35.038514 29662 consensus_queue.cc:237] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } }
I20260810 21:13:35.039422 28784 catalog_manager.cc:5495] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e326d8765d449db0afbeddd0497f4a (127.27.117.65). New cstate: current_term: 1 leader_uuid: "50e326d8765d449db0afbeddd0497f4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e326d8765d449db0afbeddd0497f4a" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 39475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 } health_report { overall_health: UNKNOWN } } }
I20260810 21:13:35.061057 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:35.058918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.27.117.65:39475)
0810 21:13:35.058979 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:35.058981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:35.058989 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:13:35.059023 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:35.059024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:35.059041 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:35.059100 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:13:35.059214 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.059650 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.060363 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:35.060533 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:35.060580 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:35.060700 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:13:35.060817 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:13:35.060910 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:35.060932 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 21:13:35.060941 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:13:35.062836 29536 raft_consensus.cc:1261] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 1 FOLLOWER]: Refusing update from remote peer 50e326d8765d449db0afbeddd0497f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:35.062820 29294 raft_consensus.cc:1261] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 1 FOLLOWER]: Refusing update from remote peer 50e326d8765d449db0afbeddd0497f4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:35.063184 29662 consensus_queue.cc:1035] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb68d613fae64494b5cecd06e673d6b4" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:35.063313 29670 consensus_queue.cc:1035] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24a217f409e45ccba042a07c1e186de" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:35.065733 29674 mvcc.cc:205] Tried to move back new op lower bound from 7317079716094586880 to 7317079715998433280. Current Snapshot: MvccSnapshot[applied={T|T < 7317079716094586880}]
I20260810 21:13:35.065925 29675 mvcc.cc:205] Tried to move back new op lower bound from 7317079716094586880 to 7317079715998433280. Current Snapshot: MvccSnapshot[applied={T|T < 7317079716094586880}]
I20260810 21:13:35.068306 28784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:35.068383 28784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:35.069068 28784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35f5269c9b4f4b2aa2dd13ade8846e9a: Sending DeleteTablet for 3 replicas of tablet 8df2d6ca9e78454796a435ec32643cbc
I20260810 21:13:35.069463 29017 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df2d6ca9e78454796a435ec32643cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:35 UTC) from {username='slave'} at 127.0.0.1:57010
I20260810 21:13:35.069495 29259 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df2d6ca9e78454796a435ec32643cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:35 UTC) from {username='slave'} at 127.0.0.1:54766
I20260810 21:13:35.069777 29691 tablet_replica.cc:330] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: stopping tablet replica
I20260810 21:13:35.069808 29690 tablet_replica.cc:330] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a: stopping tablet replica
I20260810 21:13:35.069797 29504 tablet_service.cc:1514] Processing DeleteTablet for tablet 8df2d6ca9e78454796a435ec32643cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:35 UTC) from {username='slave'} at 127.0.0.1:42634
I20260810 21:13:35.069875 29691 raft_consensus.cc:2229] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:35.069905 29690 raft_consensus.cc:2229] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:13:35.069970 29691 raft_consensus.cc:2258] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:35.070016 29690 raft_consensus.cc:2258] T 8df2d6ca9e78454796a435ec32643cbc P 50e326d8765d449db0afbeddd0497f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:35.070268 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28904
I20260810 21:13:35.073638 29691 ts_tablet_manager.cc:1905] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:35.075801 29691 ts_tablet_manager.cc:1918] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:35.075867 29691 log.cc:1196] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/8df2d6ca9e78454796a435ec32643cbc
I20260810 21:13:35.076114 29691 ts_tablet_manager.cc:1939] T 8df2d6ca9e78454796a435ec32643cbc P e24a217f409e45ccba042a07c1e186de: Deleting consensus metadata
I20260810 21:13:35.076501 29693 tablet_replica.cc:330] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: stopping tablet replica
I20260810 21:13:35.076603 29693 raft_consensus.cc:2229] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:35.076586 28759 catalog_manager.cc:4841] TS e24a217f409e45ccba042a07c1e186de (127.27.117.66:36239): tablet 8df2d6ca9e78454796a435ec32643cbc (table test-table [id=5fa373467bff4419b7a66c3b84c9b921]) successfully deleted
I20260810 21:13:35.076682 29693 raft_consensus.cc:2258] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:35.077072 29693 ts_tablet_manager.cc:1905] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:35.078457 29693 ts_tablet_manager.cc:1918] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:35.078523 29693 log.cc:1196] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/8df2d6ca9e78454796a435ec32643cbc
I20260810 21:13:35.078707 29693 ts_tablet_manager.cc:1939] T 8df2d6ca9e78454796a435ec32643cbc P bb68d613fae64494b5cecd06e673d6b4: Deleting consensus metadata
I20260810 21:13:35.079366 28752 catalog_manager.cc:4841] TS bb68d613fae64494b5cecd06e673d6b4 (127.27.117.67:42605): tablet 8df2d6ca9e78454796a435ec32643cbc (table test-table [id=5fa373467bff4419b7a66c3b84c9b921]) successfully deleted
W20260810 21:13:35.081075 28752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:39475 (error 108)
I20260810 21:13:35.081329 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29146
I20260810 21:13:35.088831 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:35.081465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:39475 (local address 127.0.0.1:57038)
0810 21:13:35.088695 (+ 7230us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:35.088756 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:39475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7200}
W20260810 21:13:35.091019 28752 catalog_manager.cc:4584] TS 50e326d8765d449db0afbeddd0497f4a (127.27.117.65:39475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8df2d6ca9e78454796a435ec32643cbc: Network error: Client connection negotiation failed: client connection to 127.27.117.65:39475: connect: Connection refused (error 111)
I20260810 21:13:35.094851 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29389
I20260810 21:13:35.106343 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 28723
2026-08-10T21:13:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (839 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:35 dist-test-slave-2kcd krb5kdc[29699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:35 dist-test-slave-2kcd krb5kdc[29699](info): set up 2 sockets
Aug 10 21:13:35 dist-test-slave-2kcd krb5kdc[29699](info): commencing operation
krb5kdc: starting...
W20260810 21:13:36.385434 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:13:36.501287 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41995
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:36.579777 29727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:36.579959 29727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:36.579980 29727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:36.581086 29727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:36.581126 29727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:36.581140 29727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:36.581153 29727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:36.581166 29727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:36.582579 29727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:36.582749 29727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:36.582999 29727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:36.587755 29738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:36.587787 29735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:36.587890 29741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:36.588124 29727 server_base.cc:1034] running on GCE node
I20260810 21:13:36.588399 29727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:36.588768 29727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:36.589979 29727 hybrid_clock.cc:648] HybridClock initialized: now 1786396416589948 us; error 41 us; skew 500 ppm
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:36.595975 29727 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:13:36.596395 29727 webserver.cc:460] Webserver started at http://127.27.117.126:36887/ using document root <none> and password file <none>
I20260810 21:13:36.596565 29727 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:36.596607 29727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:36.596694 29727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:36.597747 29727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a24c05a73134444dbfeaef33e312e1a8"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "4f2a8762dbcc31ab3a4e228032ce4177"
server_key_iv: "3ce9b1dc91ae2ccb804df13f5acae926"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.598076 29727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a24c05a73134444dbfeaef33e312e1a8"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "4f2a8762dbcc31ab3a4e228032ce4177"
server_key_iv: "3ce9b1dc91ae2ccb804df13f5acae926"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.599395 29727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:13:36.600180 29751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.600383 29727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:36.600474 29727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a24c05a73134444dbfeaef33e312e1a8"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "4f2a8762dbcc31ab3a4e228032ce4177"
server_key_iv: "3ce9b1dc91ae2ccb804df13f5acae926"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.600528 29727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:36.617646 29727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:36.620128 29727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:36.620261 29727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:36.625617 29727 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:41995
I20260810 21:13:36.625640 29843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:41995 every 8 connection(s)
I20260810 21:13:36.626097 29727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:36.627066 29844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:36.629679 29844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8: Bootstrap starting.
I20260810 21:13:36.630286 29844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:36.630527 29844 log.cc:824] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:36.631141 29844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8: No bootstrap required, opened a new log
I20260810 21:13:36.632745 29844 raft_consensus.cc:348] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24c05a73134444dbfeaef33e312e1a8" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41995 } }
I20260810 21:13:36.632946 29844 raft_consensus.cc:374] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:36.632975 29844 raft_consensus.cc:729] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24c05a73134444dbfeaef33e312e1a8, State: Initialized, Role: FOLLOWER
I20260810 21:13:36.633106 29844 consensus_queue.cc:260] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24c05a73134444dbfeaef33e312e1a8" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41995 } }
I20260810 21:13:36.633188 29844 raft_consensus.cc:388] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:36.633210 29844 raft_consensus.cc:482] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:36.633237 29844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:36.633697 29844 raft_consensus.cc:504] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24c05a73134444dbfeaef33e312e1a8" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41995 } }
I20260810 21:13:36.633811 29844 leader_election.cc:302] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a24c05a73134444dbfeaef33e312e1a8; no voters:
I20260810 21:13:36.633939 29844 leader_election.cc:288] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:36.633999 29848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:36.634172 29848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [term 1 LEADER]: Becoming Leader. State: Replica: a24c05a73134444dbfeaef33e312e1a8, State: Running, Role: LEADER
I20260810 21:13:36.634223 29844 sys_catalog.cc:564] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:36.634269 29848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24c05a73134444dbfeaef33e312e1a8" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41995 } }
I20260810 21:13:36.634634 29850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a24c05a73134444dbfeaef33e312e1a8. Latest consensus state: current_term: 1 leader_uuid: "a24c05a73134444dbfeaef33e312e1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24c05a73134444dbfeaef33e312e1a8" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41995 } } }
I20260810 21:13:36.634732 29850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:36.634763 29849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a24c05a73134444dbfeaef33e312e1a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24c05a73134444dbfeaef33e312e1a8" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41995 } } }
I20260810 21:13:36.634824 29849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:36.635443 29855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:36.635607 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29727
I20260810 21:13:36.635727 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:36.635936 28117 external_mini_cluster.cc:1428] Setting key 6500ad48f1e61b81106408aa18e46b5d
I20260810 21:13:36.635994 29855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:36.637610 29855 catalog_manager.cc:1347] Generated new cluster ID: 773c6aa76eba4316984f21cb7270694b
I20260810 21:13:36.637663 29855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:36.648252 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:36.636514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.27.117.126:41995)
0810 21:13:36.636759 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:36.636841 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:36.636878 (+ 37us) server_negotiation.cc:407] Connection header received
0810 21:13:36.636979 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:36.636984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:36.637023 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:36.637338 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:36.638139 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.639542 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.640670 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.641309 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.641716 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:36.644177 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:36.644181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:36.644181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:36.644196 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:36.646235 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:36.646634 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:36.646637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:36.646637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:36.646783 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:36.647019 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:36.647019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:36.647020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:36.647332 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:36.647826 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:36.647914 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 21:13:36.648048 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260810 21:13:36.650141 29855 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:36.650693 29855 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:36.661485 29855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8: Generated new TSK 0
I20260810 21:13:36.661695 29855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.117.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
Entry for principal kudu/127.27.117.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab.
I20260810 21:13:36.693126 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41995
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:36.774238 29890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:36.774487 29890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:36.774526 29890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:36.775671 29890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:36.775743 29890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:36.775786 29890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:36.777287 29890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.65.keytab
--principal=kudu/127.27.117.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:36.777559 29890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:36.777801 29890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:36.782816 29899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:36.782799 29902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:36.782796 29898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:36.783272 29890 server_base.cc:1034] running on GCE node
I20260810 21:13:36.783485 29890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:36.783727 29890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:36.784915 29890 hybrid_clock.cc:648] HybridClock initialized: now 1786396416784904 us; error 35 us; skew 500 ppm
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:36.787665 29890 init.cc:377] Logged in from keytab as kudu/127.27.117.65@KRBTEST.COM (short username kudu)
I20260810 21:13:36.788159 29890 webserver.cc:460] Webserver started at http://127.27.117.65:33341/ using document root <none> and password file <none>
I20260810 21:13:36.788429 29890 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:36.788481 29890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:36.788655 29890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:36.789817 29890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "73e04d3a487346dcbd88922b69d49a8c"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "6e900b82f77f33b3a55a5cdce64f428e"
server_key_iv: "bef3692c8a0145e2d60910947d20e5d5"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.790185 29890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "73e04d3a487346dcbd88922b69d49a8c"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "6e900b82f77f33b3a55a5cdce64f428e"
server_key_iv: "bef3692c8a0145e2d60910947d20e5d5"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.791549 29890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:36.792292 29913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.792474 29890 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:36.792562 29890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "73e04d3a487346dcbd88922b69d49a8c"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "6e900b82f77f33b3a55a5cdce64f428e"
server_key_iv: "bef3692c8a0145e2d60910947d20e5d5"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.792613 29890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:36.812539 29890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:36.814457 29890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:36.814570 29890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:36.814754 29890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:36.815017 29890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:36.815047 29890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.815068 29890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:36.815081 29890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.820981 29890 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:36215
I20260810 21:13:36.821060 30131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:36215 every 8 connection(s)
I20260810 21:13:36.821386 29890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.65@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:36.828279 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 29890
I20260810 21:13:36.828423 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:13:36.828649 28117 external_mini_cluster.cc:1428] Setting key 44ba21a8dd5519998f7076f6cc6568a4
I20260810 21:13:36.831192 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:36.822132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:37313 (local address 127.27.117.126:41995)
0810 21:13:36.822178 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:36.822180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:36.822396 (+ 216us) server_negotiation.cc:407] Connection header received
0810 21:13:36.822927 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:36.822929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:36.822946 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:36.822998 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:36.823970 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.824496 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.825580 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.825798 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.825889 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:36.827952 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:36.827954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:36.827954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:36.827961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:36.829657 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:36.830161 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:36.830162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:36.830162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:36.830215 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:36.830493 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:36.830493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:36.830494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:36.830593 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:36.830643 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:36.831013 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 21:13:36.831093 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:13:36.831703 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:36.822087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41995 (local address 127.27.117.65:37313)
0810 21:13:36.822275 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:36.822311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:36.822807 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:36.823081 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:36.823085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:36.823671 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:36.823868 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:36.823873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.824598 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.824600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:36.825478 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:36.825481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.825904 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.825906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:36.826042 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:36.826481 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:36.826490 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:36.827865 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:36.829741 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:36.829749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:36.829751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:36.830080 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:36.830282 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:36.830283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:36.830283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:36.830427 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:36.830658 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:36.830665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:36.830828 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 21:13:36.830954 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 21:13:36.831131 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260810 21:13:36.832229 30136 heartbeater.cc:347] Connected to a master server at 127.27.117.126:41995
I20260810 21:13:36.832338 30136 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:36.832515 30136 heartbeater.cc:511] Master 127.27.117.126:41995 requested a full tablet report, sending...
I20260810 21:13:36.833109 29774 ts_manager.cc:194] Registered new tserver with Master: 73e04d3a487346dcbd88922b69d49a8c (127.27.117.65:36215)
I20260810 21:13:36.833737 29774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.65@KRBTEST.COM'} at 127.27.117.65:37313
WARNING: no policy specified for kudu/127.27.117.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
Entry for principal kudu/127.27.117.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab.
I20260810 21:13:36.855623 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41995
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:36.937270 30146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:36.937454 30146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:36.937477 30146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:36.938598 30146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:36.938640 30146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:36.938665 30146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:36.940093 30146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.66.keytab
--principal=kudu/127.27.117.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:36.940328 30146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:36.940558 30146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:36.945356 30164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:36.945394 30166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:36.945365 30163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:36.945573 30146 server_base.cc:1034] running on GCE node
I20260810 21:13:36.945847 30146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:36.946110 30146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:36.947263 30146 hybrid_clock.cc:648] HybridClock initialized: now 1786396416947250 us; error 27 us; skew 500 ppm
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:36.949753 30146 init.cc:377] Logged in from keytab as kudu/127.27.117.66@KRBTEST.COM (short username kudu)
I20260810 21:13:36.950213 30146 webserver.cc:460] Webserver started at http://127.27.117.66:36167/ using document root <none> and password file <none>
I20260810 21:13:36.950418 30146 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:36.950480 30146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:36.950600 30146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:36.951467 30146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "bf79119c65aa44c990c51b544b73ede9"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "e47c21bba6bf03df590ebf02b693a4d1"
server_key_iv: "3b1386098b2fcb305a3bdda5241ac301"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.951750 30146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "bf79119c65aa44c990c51b544b73ede9"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "e47c21bba6bf03df590ebf02b693a4d1"
server_key_iv: "3b1386098b2fcb305a3bdda5241ac301"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.952942 30146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:13:36.953625 30176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.953814 30146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:36.953903 30146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "bf79119c65aa44c990c51b544b73ede9"
format_stamp: "Formatted at 2026-08-10 21:13:36 on dist-test-slave-2kcd"
server_key: "e47c21bba6bf03df590ebf02b693a4d1"
server_key_iv: "3b1386098b2fcb305a3bdda5241ac301"
server_key_version: "encryptionkey@0"
I20260810 21:13:36.953975 30146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:36.968405 30146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:36.970130 30146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:36.970283 30146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:36.970489 30146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:36.970840 30146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:36.970896 30146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.970938 30146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:36.970964 30146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:36.976692 30146 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:43921
I20260810 21:13:36.976749 30362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:43921 every 8 connection(s)
I20260810 21:13:36.977053 30146 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:13:36.980979 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30146
I20260810 21:13:36.981099 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:13:36.981290 28117 external_mini_cluster.cc:1428] Setting key ce560b918c9529f5732495289cb98efb
Aug 10 21:13:36 dist-test-slave-2kcd krb5kdc[29699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.66@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:36.986469 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:36.977731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:36195 (local address 127.27.117.126:41995)
0810 21:13:36.977795 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:36.977797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:36.977974 (+ 177us) server_negotiation.cc:407] Connection header received
0810 21:13:36.978399 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:36.978400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:36.978409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:36.978457 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:36.979315 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.979820 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.980808 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.981011 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.981097 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:36.983395 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:36.983396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:36.983397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:36.983412 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:36.985028 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:36.985529 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:36.985530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:36.985530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:36.985583 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:36.985850 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:36.985851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:36.985852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:36.985951 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:36.986001 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:36.986337 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 21:13:36.986407 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:13:36.986759 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:36.977677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41995 (local address 127.27.117.66:36195)
0810 21:13:36.977835 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:36.977874 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:36.978324 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:36.978511 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:36.978513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:36.979035 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:36.979200 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:36.979205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.979917 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.979918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:36.980694 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:36.980696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:36.981123 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:36.981125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:36.981281 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:36.981804 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:36.981811 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:36.983299 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:36.985113 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:36.985120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:36.985122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:36.985448 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:36.985644 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:36.985644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:36.985645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:36.985786 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:36.986015 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:36.986021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:36.986186 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 21:13:36.986323 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 21:13:36.986518 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260810 21:13:36.987280 30364 heartbeater.cc:347] Connected to a master server at 127.27.117.126:41995
I20260810 21:13:36.987366 30364 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:36.987534 30364 heartbeater.cc:511] Master 127.27.117.126:41995 requested a full tablet report, sending...
I20260810 21:13:36.987927 29774 ts_manager.cc:194] Registered new tserver with Master: bf79119c65aa44c990c51b544b73ede9 (127.27.117.66:43921)
I20260810 21:13:36.988516 29774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.66@KRBTEST.COM'} at 127.27.117.66:36195
WARNING: no policy specified for kudu/127.27.117.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
Entry for principal kudu/127.27.117.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab.
I20260810 21:13:37.007512 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41995
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:37.086709 30370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:37.086894 30370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:37.086916 30370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:37.087976 30370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:37.088008 30370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:37.088032 30370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:37.089445 30370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.67.keytab
--principal=kudu/127.27.117.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:37.089612 30370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:37.089781 30370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:37.094483 30377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.094480 30381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.094460 30386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:37.094511 30370 server_base.cc:1034] running on GCE node
I20260810 21:13:37.094796 30370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:37.095052 30370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:37.096205 30370 hybrid_clock.cc:648] HybridClock initialized: now 1786396417096206 us; error 38 us; skew 500 ppm
Aug 10 21:13:37 dist-test-slave-2kcd krb5kdc[29699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:37.098784 30370 init.cc:377] Logged in from keytab as kudu/127.27.117.67@KRBTEST.COM (short username kudu)
I20260810 21:13:37.099181 30370 webserver.cc:460] Webserver started at http://127.27.117.67:33069/ using document root <none> and password file <none>
I20260810 21:13:37.099419 30370 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:37.099486 30370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:37.099632 30370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:37.100652 30370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "4223cb7c7e6b4eec9d791333273ee1c9"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "bc2cf6c30dd89ae31adf3172aa67b7e9"
server_key_iv: "5586b3ac8a07393502e7fb05a60dc209"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.101012 30370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "4223cb7c7e6b4eec9d791333273ee1c9"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "bc2cf6c30dd89ae31adf3172aa67b7e9"
server_key_iv: "5586b3ac8a07393502e7fb05a60dc209"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.102330 30370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:37.103072 30397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.103367 30370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:37.103457 30370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "4223cb7c7e6b4eec9d791333273ee1c9"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "bc2cf6c30dd89ae31adf3172aa67b7e9"
server_key_iv: "5586b3ac8a07393502e7fb05a60dc209"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.103535 30370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:37.121690 30370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:37.123487 30370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:37.123598 30370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:37.123770 30370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:37.124002 30370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:37.124032 30370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.124051 30370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:37.124063 30370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.129969 30370 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:44287
I20260810 21:13:37.130044 30600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:44287 every 8 connection(s)
I20260810 21:13:37.130342 30370 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:37.132035 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30370
I20260810 21:13:37.132126 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:13:37.132323 28117 external_mini_cluster.cc:1428] Setting key 9606dce927f2b0c930f51b58804d9dc3
Aug 10 21:13:37 dist-test-slave-2kcd krb5kdc[29699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.67@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:37.139533 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.131070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:59875 (local address 127.27.117.126:41995)
0810 21:13:37.131119 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.131122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.131355 (+ 233us) server_negotiation.cc:407] Connection header received
0810 21:13:37.131855 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.131856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.131869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.131942 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.133161 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.133723 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.134708 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.134920 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.135020 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.137150 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.137151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.137152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:37.137159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.138375 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:37.138725 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:37.138726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:37.138726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:37.138782 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:37.139040 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:37.139041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:37.139041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:37.139145 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.139201 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.139405 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.139472 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:13:37.139715 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.131063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41995 (local address 127.27.117.67:59875)
0810 21:13:37.131240 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.131274 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.131753 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.131992 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.131995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.132771 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:37.133065 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.133072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.133796 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.133796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.134611 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.134613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.135022 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.135023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.135160 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.135655 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:13:37.135671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:37.137060 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:37.138449 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:37.138455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:37.138457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:37.138655 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:37.138864 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:13:37.138864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:13:37.138865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:13:37.138979 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:13:37.139203 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:37.139207 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:37.139311 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.139391 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.139523 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260810 21:13:37.140197 30604 heartbeater.cc:347] Connected to a master server at 127.27.117.126:41995
I20260810 21:13:37.140298 30604 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:37.140472 30604 heartbeater.cc:511] Master 127.27.117.126:41995 requested a full tablet report, sending...
I20260810 21:13:37.140914 29774 ts_manager.cc:194] Registered new tserver with Master: 4223cb7c7e6b4eec9d791333273ee1c9 (127.27.117.67:44287)
I20260810 21:13:37.141273 29774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.117.67@KRBTEST.COM'} at 127.27.117.67:59875
I20260810 21:13:37.144088 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:13:37.151674 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.145535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.27.117.126:41995)
0810 21:13:37.145602 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.145603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.145739 (+ 136us) server_negotiation.cc:407] Connection header received
0810 21:13:37.145843 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.145844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.145855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.145918 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.146525 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.146995 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.147771 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.147993 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.148085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.149265 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.149266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.149267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:37.149274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.150394 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:37.150772 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:37.150773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:37.150773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:37.150856 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:37.151079 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:37.151081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:37.151081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:37.151231 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.151349 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.151497 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.151574 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:13:37.153115 29774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:13:37.154673 29774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:13:37.161338 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.157093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:44287 (local address 127.0.0.1:45262)
0810 21:13:37.157474 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.157517 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.157599 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.157950 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.157952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.157955 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.158020 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.158021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.159294 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.159295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.160220 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.160223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.161006 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.161007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.161087 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.161102 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.161184 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.161250 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":128,"thread_start_us":75,"threads_started":1}
I20260810 21:13:37.161350 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.157201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.27.117.67:44287)
0810 21:13:37.157490 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.157495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.157578 (+ 83us) server_negotiation.cc:407] Connection header received
0810 21:13:37.157733 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.157736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.157753 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.157822 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.158097 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.159181 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.160314 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.160908 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.161004 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.161059 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.161222 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.161266 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
I20260810 21:13:37.162246 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.157733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.27.117.66:43921)
0810 21:13:37.157907 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.157912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.158183 (+ 271us) server_negotiation.cc:407] Connection header received
0810 21:13:37.158234 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.158236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.158252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.158330 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.158449 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.159420 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.160914 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.161648 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.161750 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.161816 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.162104 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.162152 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260810 21:13:37.162261 30633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.157510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39004 (local address 127.27.117.65:36215)
0810 21:13:37.157677 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.157681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.158042 (+ 361us) server_negotiation.cc:407] Connection header received
0810 21:13:37.158102 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.158104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.158122 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.158205 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.158472 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.159169 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.160667 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.161469 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.161544 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.161596 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.162134 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.162182 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260810 21:13:37.161864 30627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.157093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36215 (local address 127.0.0.1:39004)
0810 21:13:37.157941 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.157951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.158027 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.158319 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.158320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.158323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.158374 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.158375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.159282 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.159283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.160563 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.160567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.161628 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.161629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.161691 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.161695 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.161746 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.161789 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260810 21:13:37.162521 30628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.157339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43921 (local address 127.0.0.1:36618)
0810 21:13:37.158106 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.158116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.158169 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.158342 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.158343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.158344 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.158386 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.158387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.159522 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.159523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.160839 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.160842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.161756 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.161757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.161938 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.161944 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.162002 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.162052 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":62,"threads_started":1}
I20260810 21:13:37.162387 30485 tablet_service.cc:1467] Processing CreateTablet for tablet 7085564788f54f37a339b87d8b147286 (DEFAULT_TABLE table=test-table [id=66587571e2f248fb9f85c5ea960b38df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:37.162811 30485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7085564788f54f37a339b87d8b147286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.163105 30012 tablet_service.cc:1467] Processing CreateTablet for tablet 7085564788f54f37a339b87d8b147286 (DEFAULT_TABLE table=test-table [id=66587571e2f248fb9f85c5ea960b38df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:37.163224 30258 tablet_service.cc:1467] Processing CreateTablet for tablet 7085564788f54f37a339b87d8b147286 (DEFAULT_TABLE table=test-table [id=66587571e2f248fb9f85c5ea960b38df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:37.163370 30012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7085564788f54f37a339b87d8b147286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.163544 30258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7085564788f54f37a339b87d8b147286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.166215 30638 tablet_bootstrap.cc:492] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c: Bootstrap starting.
I20260810 21:13:37.166713 30639 tablet_bootstrap.cc:492] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Bootstrap starting.
I20260810 21:13:37.166904 30642 tablet_bootstrap.cc:492] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Bootstrap starting.
I20260810 21:13:37.167325 30639 tablet_bootstrap.cc:654] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.167325 30638 tablet_bootstrap.cc:654] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.167582 30642 tablet_bootstrap.cc:654] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.167907 30639 log.cc:824] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.167982 30638 log.cc:824] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.168254 30642 log.cc:824] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.168780 30638 tablet_bootstrap.cc:492] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c: No bootstrap required, opened a new log
I20260810 21:13:37.168789 30639 tablet_bootstrap.cc:492] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: No bootstrap required, opened a new log
I20260810 21:13:37.168883 30639 ts_tablet_manager.cc:1397] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:37.168891 30638 ts_tablet_manager.cc:1397] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 21:13:37.169095 30642 tablet_bootstrap.cc:492] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: No bootstrap required, opened a new log
I20260810 21:13:37.169184 30642 ts_tablet_manager.cc:1397] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:37.170836 30639 raft_consensus.cc:348] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.170836 30638 raft_consensus.cc:348] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.171026 30639 raft_consensus.cc:374] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.171026 30638 raft_consensus.cc:374] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.170928 30642 raft_consensus.cc:348] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.171066 30638 raft_consensus.cc:729] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e04d3a487346dcbd88922b69d49a8c, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.171066 30639 raft_consensus.cc:729] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4223cb7c7e6b4eec9d791333273ee1c9, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.171111 30642 raft_consensus.cc:374] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.171151 30642 raft_consensus.cc:729] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf79119c65aa44c990c51b544b73ede9, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.171190 30639 consensus_queue.cc:260] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.171252 30638 consensus_queue.cc:260] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.171280 30642 consensus_queue.cc:260] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.171530 30639 ts_tablet_manager.cc:1428] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:37.171546 30638 ts_tablet_manager.cc:1428] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 21:13:37.171602 30642 ts_tablet_manager.cc:1428] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 21:13:37.171664 30604 heartbeater.cc:503] Master 127.27.117.126:41995 was elected leader, sending a full tablet report...
I20260810 21:13:37.171727 30364 heartbeater.cc:503] Master 127.27.117.126:41995 was elected leader, sending a full tablet report...
I20260810 21:13:37.171978 30136 heartbeater.cc:503] Master 127.27.117.126:41995 was elected leader, sending a full tablet report...
W20260810 21:13:37.227596 30365 tablet.cc:2369] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:37.285727 30649 raft_consensus.cc:482] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:37.285883 30649 raft_consensus.cc:504] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.286295 30649 leader_election.cc:288] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73e04d3a487346dcbd88922b69d49a8c (127.27.117.65:36215), bf79119c65aa44c990c51b544b73ede9 (127.27.117.66:43921)
I20260810 21:13:37.290333 30633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.286648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:37749 (local address 127.27.117.65:36215)
0810 21:13:37.286699 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.286703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.286829 (+ 126us) server_negotiation.cc:407] Connection header received
0810 21:13:37.286927 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.286928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.286935 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.286979 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.287141 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.287828 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.289392 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.289979 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.290048 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.290055 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.290222 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.290262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:13:37.290333 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.286652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:59663 (local address 127.27.117.66:43921)
0810 21:13:37.286706 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.286708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.286717 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:13:37.286752 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.286753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.286760 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.286850 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.287047 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.287830 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.289392 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.289958 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.290047 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.290055 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.290222 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.290262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 21:13:37.290355 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.286472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:43921 (local address 127.27.117.67:59663)
0810 21:13:37.286536 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.286549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.286601 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.286847 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.286847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.286849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.286943 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.286944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.287919 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.287920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.289297 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.289301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.290059 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.290059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.290144 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.290147 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.290200 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.290249 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":74}
I20260810 21:13:37.290364 30652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.286474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36215 (local address 127.27.117.67:37749)
0810 21:13:37.286754 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.286762 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.286830 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.287003 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.287004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.287006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.287072 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.287073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.287919 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.287920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.289297 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.289301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.290059 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.290059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.290139 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.290143 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.290200 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.290248 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":126,"thread_start_us":136,"threads_started":1}
I20260810 21:13:37.290692 30295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7085564788f54f37a339b87d8b147286" candidate_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf79119c65aa44c990c51b544b73ede9" is_pre_election: true
I20260810 21:13:37.290840 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7085564788f54f37a339b87d8b147286" candidate_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e04d3a487346dcbd88922b69d49a8c" is_pre_election: true
I20260810 21:13:37.290894 30295 raft_consensus.cc:2454] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4223cb7c7e6b4eec9d791333273ee1c9 in term 0.
I20260810 21:13:37.291005 30051 raft_consensus.cc:2454] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4223cb7c7e6b4eec9d791333273ee1c9 in term 0.
I20260810 21:13:37.291136 30399 leader_election.cc:302] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4223cb7c7e6b4eec9d791333273ee1c9, bf79119c65aa44c990c51b544b73ede9; no voters:
I20260810 21:13:37.291253 30649 raft_consensus.cc:2781] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:13:37.291299 30649 raft_consensus.cc:482] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:13:37.291325 30649 raft_consensus.cc:3037] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:37.292017 30649 raft_consensus.cc:504] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.292161 30649 leader_election.cc:288] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [CANDIDATE]: Term 1 election: Requested vote from peers 73e04d3a487346dcbd88922b69d49a8c (127.27.117.65:36215), bf79119c65aa44c990c51b544b73ede9 (127.27.117.66:43921)
I20260810 21:13:37.292339 30051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7085564788f54f37a339b87d8b147286" candidate_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e04d3a487346dcbd88922b69d49a8c"
I20260810 21:13:37.292351 30295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7085564788f54f37a339b87d8b147286" candidate_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf79119c65aa44c990c51b544b73ede9"
I20260810 21:13:37.292433 30051 raft_consensus.cc:3037] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:37.292431 30295 raft_consensus.cc:3037] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:37.293176 30051 raft_consensus.cc:2454] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4223cb7c7e6b4eec9d791333273ee1c9 in term 1.
I20260810 21:13:37.293359 30295 raft_consensus.cc:2454] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4223cb7c7e6b4eec9d791333273ee1c9 in term 1.
I20260810 21:13:37.293404 30398 leader_election.cc:302] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4223cb7c7e6b4eec9d791333273ee1c9, 73e04d3a487346dcbd88922b69d49a8c; no voters:
I20260810 21:13:37.293507 30649 raft_consensus.cc:2781] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:37.293699 30649 raft_consensus.cc:686] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4223cb7c7e6b4eec9d791333273ee1c9, State: Running, Role: LEADER
I20260810 21:13:37.293797 30649 consensus_queue.cc:237] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } }
I20260810 21:13:37.294764 29772 catalog_manager.cc:5495] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4223cb7c7e6b4eec9d791333273ee1c9 (127.27.117.67). New cstate: current_term: 1 leader_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4223cb7c7e6b4eec9d791333273ee1c9" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 } health_report { overall_health: UNKNOWN } } }
I20260810 21:13:37.306430 30629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.304013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.27.117.67:44287)
0810 21:13:37.304069 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.304071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.304086 (+ 15us) server_negotiation.cc:407] Connection header received
0810 21:13:37.304192 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.304193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.304208 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.304265 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:13:37.304411 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.304964 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.305723 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.305904 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.305957 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.306088 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:13:37.306214 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:13:37.306276 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.306351 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.306366 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 21:13:37.308018 30051 raft_consensus.cc:1261] T 7085564788f54f37a339b87d8b147286 P 73e04d3a487346dcbd88922b69d49a8c [term 1 FOLLOWER]: Refusing update from remote peer 4223cb7c7e6b4eec9d791333273ee1c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:37.308032 30295 raft_consensus.cc:1261] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 1 FOLLOWER]: Refusing update from remote peer 4223cb7c7e6b4eec9d791333273ee1c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:37.308375 30649 consensus_queue.cc:1035] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf79119c65aa44c990c51b544b73ede9" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:37.308487 30653 consensus_queue.cc:1035] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e04d3a487346dcbd88922b69d49a8c" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:37.310530 30660 mvcc.cc:205] Tried to move back new op lower bound from 7317079725291491328 to 7317079725236199424. Current Snapshot: MvccSnapshot[applied={T|T < 7317079725291491328}]
I20260810 21:13:37.312826 29772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:37.312937 29772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:37.313675 29772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24c05a73134444dbfeaef33e312e1a8: Sending DeleteTablet for 3 replicas of tablet 7085564788f54f37a339b87d8b147286
I20260810 21:13:37.314075 30012 tablet_service.cc:1514] Processing DeleteTablet for tablet 7085564788f54f37a339b87d8b147286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:37 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:39004
I20260810 21:13:37.314064 30258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7085564788f54f37a339b87d8b147286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:37 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:36618
I20260810 21:13:37.314281 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29890
I20260810 21:13:37.314280 30485 tablet_service.cc:1514] Processing DeleteTablet for tablet 7085564788f54f37a339b87d8b147286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:37 UTC) from {username='kudu', principal='kudu/127.27.117.126@KRBTEST.COM'} at 127.0.0.1:45262
I20260810 21:13:37.322865 30684 tablet_replica.cc:330] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: stopping tablet replica
I20260810 21:13:37.323041 30684 raft_consensus.cc:2229] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:37.323186 30684 raft_consensus.cc:2258] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:37.323773 30684 ts_tablet_manager.cc:1905] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:37.325975 30684 ts_tablet_manager.cc:1918] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:37.326058 30684 log.cc:1196] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/7085564788f54f37a339b87d8b147286
I20260810 21:13:37.326318 30684 ts_tablet_manager.cc:1939] T 7085564788f54f37a339b87d8b147286 P bf79119c65aa44c990c51b544b73ede9: Deleting consensus metadata
I20260810 21:13:37.326713 29753 catalog_manager.cc:4841] TS bf79119c65aa44c990c51b544b73ede9 (127.27.117.66:43921): tablet 7085564788f54f37a339b87d8b147286 (table test-table [id=66587571e2f248fb9f85c5ea960b38df]) successfully deleted
W20260810 21:13:37.330195 29752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:36215 (error 108)
I20260810 21:13:37.330426 30685 tablet_replica.cc:330] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: stopping tablet replica
I20260810 21:13:37.330506 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30146
I20260810 21:13:37.330564 30685 raft_consensus.cc:2229] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:13:37.330685 30685 raft_consensus.cc:2258] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:37.332547 30628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.332409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:36215 (local address 127.0.0.1:39012)
0810 21:13:37.332466 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.332510 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 21:13:37.332731 29752 catalog_manager.cc:4584] TS 73e04d3a487346dcbd88922b69d49a8c (127.27.117.65:36215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7085564788f54f37a339b87d8b147286: Network error: Client connection negotiation failed: client connection to 127.27.117.65:36215: connect: Connection refused (error 111)
I20260810 21:13:37.333451 30685 ts_tablet_manager.cc:1905] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:37.335389 30685 ts_tablet_manager.cc:1918] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:37.335474 30685 log.cc:1196] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/7085564788f54f37a339b87d8b147286
I20260810 21:13:37.335784 30685 ts_tablet_manager.cc:1939] T 7085564788f54f37a339b87d8b147286 P 4223cb7c7e6b4eec9d791333273ee1c9: Deleting consensus metadata
I20260810 21:13:37.336522 29753 catalog_manager.cc:4841] TS 4223cb7c7e6b4eec9d791333273ee1c9 (127.27.117.67:44287): tablet 7085564788f54f37a339b87d8b147286 (table test-table [id=66587571e2f248fb9f85c5ea960b38df]) successfully deleted
I20260810 21:13:37.340036 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30370
I20260810 21:13:37.350051 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 29727
2026-08-10T21:13:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2241 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T21:13:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:37Z Disabled control of system clock
I20260810 21:13:37.381645 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:36933
--webserver_interface=127.27.117.126
--webserver_port=0
--builtin_ntp_servers=127.27.117.84:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:36933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:37.460734 30693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:37.460968 30693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:37.460988 30693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:37.462145 30693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:37.462185 30693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:37.462198 30693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:37.462210 30693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:37.462261 30693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:37.463636 30693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:36933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:36933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:37.463891 30693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:37.464195 30693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:37.468926 30701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.468959 30704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.468959 30702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:37.469192 30693 server_base.cc:1034] running on GCE node
I20260810 21:13:37.469576 30693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:37.469986 30693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:37.471177 30693 hybrid_clock.cc:648] HybridClock initialized: now 1786396417471157 us; error 33 us; skew 500 ppm
I20260810 21:13:37.472314 30693 webserver.cc:460] Webserver started at http://127.27.117.126:35579/ using document root <none> and password file <none>
I20260810 21:13:37.472558 30693 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:37.472627 30693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:37.472815 30693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:37.473830 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "892f459aaf0c4fae9abbec96964d4833"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "4d152f531e4617cdaabdd72315fa808d"
server_key_iv: "be8db69be2f921405be18b93a0404fe6"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.474179 30693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "892f459aaf0c4fae9abbec96964d4833"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "4d152f531e4617cdaabdd72315fa808d"
server_key_iv: "be8db69be2f921405be18b93a0404fe6"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.475476 30693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:13:37.476115 30718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.476274 30693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:37.476387 30693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "892f459aaf0c4fae9abbec96964d4833"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "4d152f531e4617cdaabdd72315fa808d"
server_key_iv: "be8db69be2f921405be18b93a0404fe6"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.476465 30693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:37.495393 30693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:37.495716 30693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:37.495875 30693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:37.499969 30693 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:36933
I20260810 21:13:37.500000 30814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:36933 every 8 connection(s)
I20260810 21:13:37.500320 30693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:37.501286 30815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.503319 30815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833: Bootstrap starting.
I20260810 21:13:37.503818 30815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.504065 30815 log.cc:824] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.504647 30815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833: No bootstrap required, opened a new log
I20260810 21:13:37.505874 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30693
I20260810 21:13:37.506021 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:37.506208 28117 external_mini_cluster.cc:1428] Setting key 673f0579346c3de78097fd093fd0aaa7
I20260810 21:13:37.506418 30815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892f459aaf0c4fae9abbec96964d4833" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36933 } }
I20260810 21:13:37.506583 30815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.506623 30815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892f459aaf0c4fae9abbec96964d4833, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.506734 30815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892f459aaf0c4fae9abbec96964d4833" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36933 } }
I20260810 21:13:37.506829 30815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:37.506871 30815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:37.506922 30815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:37.507449 30815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892f459aaf0c4fae9abbec96964d4833" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36933 } }
I20260810 21:13:37.507601 30815 leader_election.cc:302] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892f459aaf0c4fae9abbec96964d4833; no voters:
I20260810 21:13:37.507756 30815 leader_election.cc:288] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:37.507827 30824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:37.507972 30824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [term 1 LEADER]: Becoming Leader. State: Replica: 892f459aaf0c4fae9abbec96964d4833, State: Running, Role: LEADER
I20260810 21:13:37.508036 30815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:37.508055 30824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892f459aaf0c4fae9abbec96964d4833" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36933 } }
I20260810 21:13:37.508390 30825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892f459aaf0c4fae9abbec96964d4833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892f459aaf0c4fae9abbec96964d4833" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36933 } } }
I20260810 21:13:37.508466 30825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:37.508714 30827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892f459aaf0c4fae9abbec96964d4833. Latest consensus state: current_term: 1 leader_uuid: "892f459aaf0c4fae9abbec96964d4833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892f459aaf0c4fae9abbec96964d4833" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36933 } } }
I20260810 21:13:37.509155 30838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:37.509754 30838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:37.509825 30827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:37.512560 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.506675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.27.117.126:36933)
0810 21:13:37.506947 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.506956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.506992 (+ 36us) server_negotiation.cc:407] Connection header received
0810 21:13:37.507166 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.507171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.507230 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.507363 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.507614 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.509454 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.510551 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.511092 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.511234 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.511916 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.511919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.511920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:37.512072 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.512102 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:37.512112 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.512164 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.512263 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.512343 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":120,"threads_started":1}
I20260810 21:13:37.517678 30838 catalog_manager.cc:1347] Generated new cluster ID: 2a088f98ba6545588c6d50e9c18de411
I20260810 21:13:37.517757 30838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:37.525252 30838 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:37.525916 30838 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:37.533342 30838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833: Generated new TSK 0
I20260810 21:13:37.533504 30838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:37.537073 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.65:0
--local_ip_for_outbound_sockets=127.27.117.65
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:36933
--builtin_ntp_servers=127.27.117.84:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:37.620390 30862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:37.620584 30862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:37.620604 30862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:37.621766 30862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:37.621809 30862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:37.621832 30862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.65
I20260810 21:13:37.623672 30862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.117.65
--webserver_port=0
--tserver_master_addrs=127.27.117.126:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.65
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:37.623919 30862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:37.624269 30862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:37.629638 30872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.629704 30873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.629684 30877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:37.630069 30862 server_base.cc:1034] running on GCE node
I20260810 21:13:37.630270 30862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:37.630483 30862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:37.631635 30862 hybrid_clock.cc:648] HybridClock initialized: now 1786396417631620 us; error 31 us; skew 500 ppm
I20260810 21:13:37.633036 30862 webserver.cc:460] Webserver started at http://127.27.117.65:41087/ using document root <none> and password file <none>
I20260810 21:13:37.633287 30862 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:37.633332 30862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:37.633481 30862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:37.634531 30862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data/instance:
uuid: "4f77eb5514bf4ece9dba52c270d10ed9"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "cea3cdaffd9b3e7ca11a3d08c4caf1c4"
server_key_iv: "51edf149038c9d1a92b586aa5b8fdaaf"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.634894 30862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance:
uuid: "4f77eb5514bf4ece9dba52c270d10ed9"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "cea3cdaffd9b3e7ca11a3d08c4caf1c4"
server_key_iv: "51edf149038c9d1a92b586aa5b8fdaaf"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.636233 30862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:37.637044 30890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.637230 30862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:37.637327 30862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal
uuid: "4f77eb5514bf4ece9dba52c270d10ed9"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "cea3cdaffd9b3e7ca11a3d08c4caf1c4"
server_key_iv: "51edf149038c9d1a92b586aa5b8fdaaf"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.637400 30862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:37.666302 30862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:37.666674 30862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:37.666850 30862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:37.667116 30862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:37.667479 30862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:37.667512 30862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.667557 30862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:37.667585 30862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.673787 30862 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.65:34781
I20260810 21:13:37.673842 31101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.65:34781 every 8 connection(s)
I20260810 21:13:37.674170 30862 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/data/info.pb
I20260810 21:13:37.679703 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.674929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:43443 (local address 127.27.117.126:36933)
0810 21:13:37.674979 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.674981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.675189 (+ 208us) server_negotiation.cc:407] Connection header received
0810 21:13:37.675742 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.675743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.675759 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.675824 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.676525 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.677120 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.678107 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.678310 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.678406 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.679363 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.679364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.679365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:37.679373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.679398 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:37.679403 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.679474 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.679578 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.679617 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:13:37.679996 31109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.674867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36933 (local address 127.27.117.65:43443)
0810 21:13:37.675062 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.675098 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.675624 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.675905 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.675909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.675925 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:37.676422 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.676426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.677219 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.677220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.677995 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.677997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.678417 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.678418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.678553 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.679191 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:37.679192 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:37.679221 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:37.679246 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:37.679473 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:37.679481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:37.679484 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.679579 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.679686 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260810 21:13:37.680610 31104 heartbeater.cc:347] Connected to a master server at 127.27.117.126:36933
I20260810 21:13:37.680721 31104 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:37.680941 31104 heartbeater.cc:511] Master 127.27.117.126:36933 requested a full tablet report, sending...
I20260810 21:13:37.681494 30744 ts_manager.cc:194] Registered new tserver with Master: 4f77eb5514bf4ece9dba52c270d10ed9 (127.27.117.65:34781)
I20260810 21:13:37.682116 30744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.65:43443
I20260810 21:13:37.683135 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 30862
I20260810 21:13:37.683220 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-0/wal/instance
I20260810 21:13:37.683368 28117 external_mini_cluster.cc:1428] Setting key e489e785d7b114568b301722eee0dbee
I20260810 21:13:37.684283 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.66:0
--local_ip_for_outbound_sockets=127.27.117.66
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:36933
--builtin_ntp_servers=127.27.117.84:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:37.767706 31112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:37.767900 31112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:37.767925 31112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:37.769085 31112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:37.769124 31112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:37.769150 31112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.66
I20260810 21:13:37.770522 31112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.117.66
--webserver_port=0
--tserver_master_addrs=127.27.117.126:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.66
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:37.770706 31112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:37.770941 31112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:37.776005 31117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.776012 31120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.776012 31122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:37.776333 31112 server_base.cc:1034] running on GCE node
I20260810 21:13:37.776543 31112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:37.776829 31112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:37.778028 31112 hybrid_clock.cc:648] HybridClock initialized: now 1786396417778016 us; error 28 us; skew 500 ppm
I20260810 21:13:37.779371 31112 webserver.cc:460] Webserver started at http://127.27.117.66:46297/ using document root <none> and password file <none>
I20260810 21:13:37.779670 31112 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:37.779718 31112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:37.779867 31112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:37.780956 31112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data/instance:
uuid: "f8a8d0964cdd48f7af894efb7abe07f2"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "da299bdf765b624abadaa77117dcb7d6"
server_key_iv: "042600dcac0e66398d6f4eca35fdd2cb"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.781294 31112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance:
uuid: "f8a8d0964cdd48f7af894efb7abe07f2"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "da299bdf765b624abadaa77117dcb7d6"
server_key_iv: "042600dcac0e66398d6f4eca35fdd2cb"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.782567 31112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:37.783295 31134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.783478 31112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:37.783571 31112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal
uuid: "f8a8d0964cdd48f7af894efb7abe07f2"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "da299bdf765b624abadaa77117dcb7d6"
server_key_iv: "042600dcac0e66398d6f4eca35fdd2cb"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.783644 31112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:37.804450 31112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:37.804733 31112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:37.804919 31112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:37.805148 31112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:37.805465 31112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:37.805498 31112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.805519 31112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:37.805564 31112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.811976 31112 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.66:46207
I20260810 21:13:37.812028 31337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.66:46207 every 8 connection(s)
I20260810 21:13:37.812335 31112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/data/info.pb
I20260810 21:13:37.817328 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.813097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.66:59997 (local address 127.27.117.126:36933)
0810 21:13:37.813166 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.813168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.813382 (+ 214us) server_negotiation.cc:407] Connection header received
0810 21:13:37.813855 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.813855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.813865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.813939 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.814614 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.815097 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.816050 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.816232 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.816283 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.817099 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.817101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.817101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:37.817105 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.817118 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:37.817122 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.817166 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.817247 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.817271 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 21:13:37.817554 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.813061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36933 (local address 127.27.117.66:59997)
0810 21:13:37.813248 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.813288 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.813771 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.813995 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.813998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.814005 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:37.814543 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.814547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.815183 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.815183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.815976 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.815978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.816314 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.816315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.816426 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.816973 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:37.816973 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:37.816999 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:37.817020 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:37.817179 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:37.817183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:37.817185 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.817236 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.817311 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260810 21:13:37.818116 31345 heartbeater.cc:347] Connected to a master server at 127.27.117.126:36933
I20260810 21:13:37.818202 31345 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:37.818331 31345 heartbeater.cc:511] Master 127.27.117.126:36933 requested a full tablet report, sending...
I20260810 21:13:37.818698 30744 ts_manager.cc:194] Registered new tserver with Master: f8a8d0964cdd48f7af894efb7abe07f2 (127.27.117.66:46207)
I20260810 21:13:37.819025 30744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.66:59997
I20260810 21:13:37.820029 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31112
I20260810 21:13:37.820114 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal/instance
I20260810 21:13:37.820263 28117 external_mini_cluster.cc:1428] Setting key f003b1f55c71486090f08d5b3df69dfc
I20260810 21:13:37.821185 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.117.67:0
--local_ip_for_outbound_sockets=127.27.117.67
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:36933
--builtin_ntp_servers=127.27.117.84:43991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 21:13:37.903585 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:37.903810 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:37.903846 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:37.904963 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:37.905023 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:37.905062 31351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.117.67
I20260810 21:13:37.906458 31351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:43991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.117.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.117.67
--webserver_port=0
--tserver_master_addrs=127.27.117.126:36933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.117.67
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:37.906711 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:37.907001 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:37.912222 31364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:37.912240 31366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:37.912286 31351 server_base.cc:1034] running on GCE node
W20260810 21:13:37.912236 31361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:37.912599 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:37.912920 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:37.914088 31351 hybrid_clock.cc:648] HybridClock initialized: now 1786396417914076 us; error 35 us; skew 500 ppm
I20260810 21:13:37.915273 31351 webserver.cc:460] Webserver started at http://127.27.117.67:42431/ using document root <none> and password file <none>
I20260810 21:13:37.915520 31351 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:37.915589 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:37.915725 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:37.916736 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data/instance:
uuid: "d59cda991ee3429fb53324334ad2292f"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "b51f79cf869b71f3dae183600b041d1f"
server_key_iv: "d83eea418f966093cdab9a5e6752eb49"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.917194 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance:
uuid: "d59cda991ee3429fb53324334ad2292f"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "b51f79cf869b71f3dae183600b041d1f"
server_key_iv: "d83eea418f966093cdab9a5e6752eb49"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.918534 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:37.919341 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.919531 31351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:37.919642 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal
uuid: "d59cda991ee3429fb53324334ad2292f"
format_stamp: "Formatted at 2026-08-10 21:13:37 on dist-test-slave-2kcd"
server_key: "b51f79cf869b71f3dae183600b041d1f"
server_key_iv: "d83eea418f966093cdab9a5e6752eb49"
server_key_version: "encryptionkey@0"
I20260810 21:13:37.919730 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:37.942009 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:37.942328 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:37.942495 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:37.942749 31351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 21:13:37.943110 31351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 21:13:37.943172 31351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.943219 31351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 21:13:37.943269 31351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:37.949797 31351 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.67:44051
I20260810 21:13:37.949859 31595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.67:44051 every 8 connection(s)
I20260810 21:13:37.950197 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/data/info.pb
I20260810 21:13:37.955150 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.950919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.67:39891 (local address 127.27.117.126:36933)
0810 21:13:37.951000 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.951002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.951230 (+ 228us) server_negotiation.cc:407] Connection header received
0810 21:13:37.951696 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.951697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.951708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.951775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.952404 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.952934 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.953882 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.954067 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.954127 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.954920 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.954922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.954922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:37.954926 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.954941 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:37.954945 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.954999 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.955062 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.955087 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:13:37.955365 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.950905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36933 (local address 127.27.117.67:39891)
0810 21:13:37.951099 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.951139 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.951610 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.951823 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.951827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.951836 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:13:37.952318 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.952322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.953018 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.953018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.953800 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.953802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.954162 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.954164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.954275 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.954785 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 21:13:37.954785 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:13:37.954812 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 21:13:37.954833 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:13:37.955001 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:13:37.955003 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:13:37.955005 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.955049 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.955125 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260810 21:13:37.955902 31596 heartbeater.cc:347] Connected to a master server at 127.27.117.126:36933
I20260810 21:13:37.956008 31596 heartbeater.cc:464] Registering TS with master...
I20260810 21:13:37.956195 31596 heartbeater.cc:511] Master 127.27.117.126:36933 requested a full tablet report, sending...
I20260810 21:13:37.956544 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31351
I20260810 21:13:37.956632 30744 ts_manager.cc:194] Registered new tserver with Master: d59cda991ee3429fb53324334ad2292f (127.27.117.67:44051)
I20260810 21:13:37.956672 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal/instance
I20260810 21:13:37.956903 28117 external_mini_cluster.cc:1428] Setting key 9f3553e5acb15bd9f0cba94a212e3735
I20260810 21:13:37.957095 30744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.117.67:39891
I20260810 21:13:37.957823 28117 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 21:13:37.961697 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.959146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.27.117.126:36933)
0810 21:13:37.959206 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.959208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.959311 (+ 103us) server_negotiation.cc:407] Connection header received
0810 21:13:37.959360 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.959361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.959370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.959421 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:37.959561 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.959990 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.960687 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.960929 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.960979 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.961533 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:37.961534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:37.961534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:37.961538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:37.961548 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:37.961551 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:37.961597 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.961631 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.961649 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 21:13:37.963086 30744 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 21:13:37.964646 30744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 21:13:37.970682 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.966918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.27.117.67:44051)
0810 21:13:37.967481 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.967487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.967496 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:13:37.967533 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.967535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.967549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.967623 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.967747 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.968385 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.969590 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.970289 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.970390 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.970468 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.970577 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.970610 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":78,"threads_started":1}
I20260810 21:13:37.970683 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.966727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:44051 (local address 127.0.0.1:38670)
0810 21:13:37.966830 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.966881 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.966952 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.967621 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.967622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.967624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.967670 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.967671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.968505 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.968506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.969520 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.969523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.970399 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.970400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.970478 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.970491 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.970562 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.970612 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":24,"thread_start_us":56,"threads_started":1}
I20260810 21:13:37.971352 31607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.966780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:34781 (local address 127.0.0.1:42634)
0810 21:13:37.967171 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.967182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.967245 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.967565 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.967568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.967570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.967631 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.967633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.968468 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.968469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.970265 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.970268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.971134 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.971135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.971192 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.971195 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.971245 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.971282 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":44,"thread_start_us":99,"threads_started":1}
I20260810 21:13:37.971370 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.966947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.27.117.65:34781)
0810 21:13:37.967143 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.967147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.967307 (+ 160us) server_negotiation.cc:407] Connection header received
0810 21:13:37.967354 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.967356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.967372 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.967472 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.967717 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.968373 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.970362 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.971034 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.971130 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.971214 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.971260 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.971294 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260810 21:13:37.971588 31462 tablet_service.cc:1467] Processing CreateTablet for tablet 0e61d080df224e6a92e54441adaabf0f (DEFAULT_TABLE table=test-table [id=f1ed92b12b96445682080c6165501754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:37.971958 31462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61d080df224e6a92e54441adaabf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.972033 30983 tablet_service.cc:1467] Processing CreateTablet for tablet 0e61d080df224e6a92e54441adaabf0f (DEFAULT_TABLE table=test-table [id=f1ed92b12b96445682080c6165501754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:37.972404 30983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61d080df224e6a92e54441adaabf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.973697 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.966943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:46207 (local address 127.0.0.1:33196)
0810 21:13:37.967234 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:37.967244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:37.967296 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:37.968675 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:37.968676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:37.968678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:37.968721 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.968723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.969661 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.969662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.970741 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:37.970745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.973467 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.973468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:37.973530 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.973534 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:37.973591 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 21:13:37.973626 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"thread_start_us":276,"threads_started":1}
I20260810 21:13:37.973992 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:37.967175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.27.117.66:46207)
0810 21:13:37.968509 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:37.968512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:37.968520 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:13:37.968550 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:37.968567 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:37.968578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:37.968677 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:37.968798 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.969569 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.970849 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:37.973275 (+ 2426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:37.973347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:37.973428 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:37.973859 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 21:13:37.973918 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":105,"threads_started":1}
I20260810 21:13:37.974862 31619 tablet_bootstrap.cc:492] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Bootstrap starting.
I20260810 21:13:37.975441 31619 tablet_bootstrap.cc:654] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.975688 31620 tablet_bootstrap.cc:492] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: Bootstrap starting.
I20260810 21:13:37.975776 31619 log.cc:824] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.976428 31620 tablet_bootstrap.cc:654] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.976410 31215 tablet_service.cc:1467] Processing CreateTablet for tablet 0e61d080df224e6a92e54441adaabf0f (DEFAULT_TABLE table=test-table [id=f1ed92b12b96445682080c6165501754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 21:13:37.976727 31619 tablet_bootstrap.cc:492] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: No bootstrap required, opened a new log
I20260810 21:13:37.976756 31215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e61d080df224e6a92e54441adaabf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:37.976784 31620 log.cc:824] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.976830 31619 ts_tablet_manager.cc:1397] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:37.977681 31620 tablet_bootstrap.cc:492] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: No bootstrap required, opened a new log
I20260810 21:13:37.977763 31620 ts_tablet_manager.cc:1397] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 21:13:37.979411 31619 raft_consensus.cc:348] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:37.979588 31619 raft_consensus.cc:374] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.979631 31619 raft_consensus.cc:729] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d59cda991ee3429fb53324334ad2292f, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.979758 31619 consensus_queue.cc:260] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:37.979813 31623 tablet_bootstrap.cc:492] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Bootstrap starting.
I20260810 21:13:37.980029 31619 ts_tablet_manager.cc:1428] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 21:13:37.980085 31596 heartbeater.cc:503] Master 127.27.117.126:36933 was elected leader, sending a full tablet report...
I20260810 21:13:37.980258 31620 raft_consensus.cc:348] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:37.980436 31620 raft_consensus.cc:374] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.980484 31620 raft_consensus.cc:729] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f77eb5514bf4ece9dba52c270d10ed9, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.980600 31620 consensus_queue.cc:260] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:37.980700 31623 tablet_bootstrap.cc:654] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:37.980903 31620 ts_tablet_manager.cc:1428] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260810 21:13:37.980927 31104 heartbeater.cc:503] Master 127.27.117.126:36933 was elected leader, sending a full tablet report...
I20260810 21:13:37.981047 31623 log.cc:824] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:37.981822 31623 tablet_bootstrap.cc:492] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: No bootstrap required, opened a new log
I20260810 21:13:37.981899 31623 ts_tablet_manager.cc:1397] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 21:13:37.983618 31623 raft_consensus.cc:348] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:37.983728 31623 raft_consensus.cc:374] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:37.983749 31623 raft_consensus.cc:729] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a8d0964cdd48f7af894efb7abe07f2, State: Initialized, Role: FOLLOWER
I20260810 21:13:37.983815 31623 consensus_queue.cc:260] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:37.984021 31623 ts_tablet_manager.cc:1428] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 21:13:37.984127 31345 heartbeater.cc:503] Master 127.27.117.126:36933 was elected leader, sending a full tablet report...
W20260810 21:13:38.063009 31349 tablet.cc:2369] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 21:13:38.174989 31108 tablet.cc:2369] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:38.189502 31625 raft_consensus.cc:482] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 21:13:38.189604 31625 raft_consensus.cc:504] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:38.190032 31625 leader_election.cc:288] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d59cda991ee3429fb53324334ad2292f (127.27.117.67:44051), f8a8d0964cdd48f7af894efb7abe07f2 (127.27.117.66:46207)
I20260810 21:13:38.192993 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:38.190289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:56095 (local address 127.27.117.67:44051)
0810 21:13:38.190359 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:38.190360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:38.190387 (+ 27us) server_negotiation.cc:407] Connection header received
0810 21:13:38.190446 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:38.190447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:38.190456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:38.190526 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:38.190709 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.191237 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.192183 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.192557 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.192630 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:38.192636 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:38.192863 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 21:13:38.192919 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 21:13:38.193037 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:38.190480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.117.65:52425 (local address 127.27.117.66:46207)
0810 21:13:38.190594 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:38.190596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:38.190605 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:13:38.190629 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:38.190630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:38.190642 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:38.190691 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 21:13:38.190874 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.191364 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.192259 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.192652 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.192723 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:38.192731 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:38.192922 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 21:13:38.192969 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 21:13:38.193055 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:38.190203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.67:44051 (local address 127.27.117.65:56095)
0810 21:13:38.190313 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:38.190325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:38.190380 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:38.190546 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:38.190547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:38.190551 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:38.190629 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:38.190631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.191316 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.191317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:38.192095 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:38.192097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.192656 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.192657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:38.192736 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:38.192741 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 21:13:38.192858 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 21:13:38.192918 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260810 21:13:38.193058 31630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:38.190378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.66:46207 (local address 127.27.117.65:52425)
0810 21:13:38.190474 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:38.190483 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 21:13:38.190519 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:13:38.190719 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:13:38.190720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:13:38.190721 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 21:13:38.190780 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:38.190782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.191446 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.191447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:38.192166 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:13:38.192169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.192735 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.192736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:13:38.192836 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:38.192840 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 21:13:38.192899 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 21:13:38.192934 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
I20260810 21:13:38.193432 31491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61d080df224e6a92e54441adaabf0f" candidate_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59cda991ee3429fb53324334ad2292f" is_pre_election: true
I20260810 21:13:38.193603 31491 raft_consensus.cc:2454] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f77eb5514bf4ece9dba52c270d10ed9 in term 0.
I20260810 21:13:38.193554 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61d080df224e6a92e54441adaabf0f" candidate_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" is_pre_election: true
I20260810 21:13:38.193720 31254 raft_consensus.cc:2454] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f77eb5514bf4ece9dba52c270d10ed9 in term 0.
I20260810 21:13:38.193857 30895 leader_election.cc:302] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f77eb5514bf4ece9dba52c270d10ed9, d59cda991ee3429fb53324334ad2292f; no voters:
I20260810 21:13:38.193994 31625 raft_consensus.cc:2781] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 21:13:38.194079 31625 raft_consensus.cc:482] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 21:13:38.194109 31625 raft_consensus.cc:3037] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:38.194829 31625 raft_consensus.cc:504] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:38.194967 31625 leader_election.cc:288] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [CANDIDATE]: Term 1 election: Requested vote from peers d59cda991ee3429fb53324334ad2292f (127.27.117.67:44051), f8a8d0964cdd48f7af894efb7abe07f2 (127.27.117.66:46207)
I20260810 21:13:38.195147 31491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61d080df224e6a92e54441adaabf0f" candidate_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d59cda991ee3429fb53324334ad2292f"
I20260810 21:13:38.195166 31254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e61d080df224e6a92e54441adaabf0f" candidate_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8a8d0964cdd48f7af894efb7abe07f2"
I20260810 21:13:38.195245 31491 raft_consensus.cc:3037] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:38.195242 31254 raft_consensus.cc:3037] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:38.195963 31254 raft_consensus.cc:2454] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f77eb5514bf4ece9dba52c270d10ed9 in term 1.
I20260810 21:13:38.196036 31491 raft_consensus.cc:2454] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f77eb5514bf4ece9dba52c270d10ed9 in term 1.
I20260810 21:13:38.196151 30895 leader_election.cc:302] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f77eb5514bf4ece9dba52c270d10ed9, f8a8d0964cdd48f7af894efb7abe07f2; no voters:
I20260810 21:13:38.196262 31625 raft_consensus.cc:2781] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:38.196424 31625 raft_consensus.cc:686] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 4f77eb5514bf4ece9dba52c270d10ed9, State: Running, Role: LEADER
I20260810 21:13:38.196530 31625 consensus_queue.cc:237] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } }
I20260810 21:13:38.197371 30744 catalog_manager.cc:5495] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f77eb5514bf4ece9dba52c270d10ed9 (127.27.117.65). New cstate: current_term: 1 leader_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f77eb5514bf4ece9dba52c270d10ed9" member_type: VOTER last_known_addr { host: "127.27.117.65" port: 34781 } health_report { overall_health: HEALTHY } } }
W20260810 21:13:38.200690 31597 tablet.cc:2369] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:13:38.253036 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:38.249415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.27.117.65:34781)
0810 21:13:38.249509 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:38.249511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:38.249524 (+ 13us) server_negotiation.cc:407] Connection header received
0810 21:13:38.249576 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:38.249577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:38.249606 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:38.249690 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:13:38.249872 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.250689 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.251964 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:38.252306 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:38.252403 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:38.252582 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:13:38.252734 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:13:38.252865 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:38.252888 (+ 23us) server_negotiation.cc:299] Negotiation successful
0810 21:13:38.252927 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 21:13:38.255412 31254 raft_consensus.cc:1261] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 1 FOLLOWER]: Refusing update from remote peer 4f77eb5514bf4ece9dba52c270d10ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:38.255478 31491 raft_consensus.cc:1261] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 1 FOLLOWER]: Refusing update from remote peer 4f77eb5514bf4ece9dba52c270d10ed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:13:38.255848 31631 consensus_queue.cc:1035] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8a8d0964cdd48f7af894efb7abe07f2" member_type: VOTER last_known_addr { host: "127.27.117.66" port: 46207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:38.256081 31631 consensus_queue.cc:1035] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d59cda991ee3429fb53324334ad2292f" member_type: VOTER last_known_addr { host: "127.27.117.67" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:13:38.260742 31636 mvcc.cc:205] Tried to move back new op lower bound from 7317079729170980864 to 7317079728933535744. Current Snapshot: MvccSnapshot[applied={T|T < 7317079729170980864}]
I20260810 21:13:38.260872 30744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:38.260959 30744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 21:13:38.261729 30744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 892f459aaf0c4fae9abbec96964d4833: Sending DeleteTablet for 3 replicas of tablet 0e61d080df224e6a92e54441adaabf0f
I20260810 21:13:38.262091 31462 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e61d080df224e6a92e54441adaabf0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:38 UTC) from {username='slave'} at 127.0.0.1:38670
I20260810 21:13:38.262133 31215 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e61d080df224e6a92e54441adaabf0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:38 UTC) from {username='slave'} at 127.0.0.1:33196
I20260810 21:13:38.262156 30983 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e61d080df224e6a92e54441adaabf0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 21:13:38 UTC) from {username='slave'} at 127.0.0.1:42634
I20260810 21:13:38.262363 31647 tablet_replica.cc:330] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: stopping tablet replica
I20260810 21:13:38.262388 31648 tablet_replica.cc:330] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9: stopping tablet replica
I20260810 21:13:38.262460 31647 raft_consensus.cc:2229] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:38.262475 31648 raft_consensus.cc:2229] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 21:13:38.262565 31647 raft_consensus.cc:2258] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:38.262557 31648 raft_consensus.cc:2258] T 0e61d080df224e6a92e54441adaabf0f P 4f77eb5514bf4ece9dba52c270d10ed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:38.262770 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30862
I20260810 21:13:38.263018 31647 ts_tablet_manager.cc:1905] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:38.271181 31647 ts_tablet_manager.cc:1918] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:38.271198 31646 tablet_replica.cc:330] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: stopping tablet replica
I20260810 21:13:38.271278 31647 log.cc:1196] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-1/wal/wals/0e61d080df224e6a92e54441adaabf0f
I20260810 21:13:38.271327 31646 raft_consensus.cc:2229] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 21:13:38.271462 31646 raft_consensus.cc:2258] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 21:13:38.271544 31647 ts_tablet_manager.cc:1939] T 0e61d080df224e6a92e54441adaabf0f P f8a8d0964cdd48f7af894efb7abe07f2: Deleting consensus metadata
I20260810 21:13:38.271997 30727 catalog_manager.cc:4841] TS f8a8d0964cdd48f7af894efb7abe07f2 (127.27.117.66:46207): tablet 0e61d080df224e6a92e54441adaabf0f (table test-table [id=f1ed92b12b96445682080c6165501754]) successfully deleted
I20260810 21:13:38.272048 31646 ts_tablet_manager.cc:1905] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 21:13:38.273447 31646 ts_tablet_manager.cc:1918] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 21:13:38.273540 31646 log.cc:1196] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Deleting WAL directory at /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786396166878326-28117-0/minicluster-data/ts-2/wal/wals/0e61d080df224e6a92e54441adaabf0f
I20260810 21:13:38.273763 31646 ts_tablet_manager.cc:1939] T 0e61d080df224e6a92e54441adaabf0f P d59cda991ee3429fb53324334ad2292f: Deleting consensus metadata
I20260810 21:13:38.274169 30727 catalog_manager.cc:4841] TS d59cda991ee3429fb53324334ad2292f (127.27.117.67:44051): tablet 0e61d080df224e6a92e54441adaabf0f (table test-table [id=f1ed92b12b96445682080c6165501754]) successfully deleted
W20260810 21:13:38.276557 30719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.117.65:34781 (error 108)
I20260810 21:13:38.276813 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31112
I20260810 21:13:38.283469 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:38.276987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.65:34781 (local address 127.0.0.1:42650)
0810 21:13:38.283366 (+ 6379us) negotiation.cc:107] Waiting for socket to connect
0810 21:13:38.283426 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.65:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6349}
W20260810 21:13:38.285888 30719 catalog_manager.cc:4584] TS 4f77eb5514bf4ece9dba52c270d10ed9 (127.27.117.65:34781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e61d080df224e6a92e54441adaabf0f: Network error: Client connection negotiation failed: client connection to 127.27.117.65:34781: connect: Connection refused (error 111)
I20260810 21:13:38.290287 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31351
I20260810 21:13:38.309253 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 30693
2026-08-10T21:13:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (965 ms)
[----------] 6 tests from EncryptionPolicyTest (9524 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:38 dist-test-slave-2kcd krb5kdc[31660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:38 dist-test-slave-2kcd krb5kdc[31660](info): set up 2 sockets
Aug 10 21:13:38 dist-test-slave-2kcd krb5kdc[31660](info): commencing operation
krb5kdc: starting...
W20260810 21:13:39.595996 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:39 dist-test-slave-2kcd krb5kdc[31660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:39.720880 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:39.800606 31681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:39.800807 31681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:39.800830 31681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:39.801851 31681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:39.801883 31681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:39.801898 31681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:39.801909 31681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:39.801921 31681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:39.803321 31681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:39.803525 31681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:39.803679 31681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:39.808322 31691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:39.808343 31693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:39.808576 31681 server_base.cc:1034] running on GCE node
W20260810 21:13:39.808305 31689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:39.808946 31681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:39.809275 31681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:39.810415 31681 hybrid_clock.cc:648] HybridClock initialized: now 1786396419810403 us; error 30 us; skew 500 ppm
Aug 10 21:13:39 dist-test-slave-2kcd krb5kdc[31660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:39.812974 31681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:39.813381 31681 webserver.cc:460] Webserver started at http://127.0.0.1:44707/ using document root <none> and password file <none>
I20260810 21:13:39.813628 31681 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:39.813696 31681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:39.813838 31681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:39.814790 31681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "7e328d06ab3241b3abd5561ce4aa30f2"
format_stamp: "Formatted at 2026-08-10 21:13:39 on dist-test-slave-2kcd"
server_key: "a620e6185ce60322a4b9c666ff32c022"
server_key_iv: "38c69802dd84ace655d3cf6648a72f1f"
server_key_version: "encryptionkey@0"
I20260810 21:13:39.815148 31681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "7e328d06ab3241b3abd5561ce4aa30f2"
format_stamp: "Formatted at 2026-08-10 21:13:39 on dist-test-slave-2kcd"
server_key: "a620e6185ce60322a4b9c666ff32c022"
server_key_iv: "38c69802dd84ace655d3cf6648a72f1f"
server_key_version: "encryptionkey@0"
I20260810 21:13:39.816460 31681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:13:39.817313 31705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:39.817505 31681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:13:39.817616 31681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "7e328d06ab3241b3abd5561ce4aa30f2"
format_stamp: "Formatted at 2026-08-10 21:13:39 on dist-test-slave-2kcd"
server_key: "a620e6185ce60322a4b9c666ff32c022"
server_key_iv: "38c69802dd84ace655d3cf6648a72f1f"
server_key_version: "encryptionkey@0"
I20260810 21:13:39.817690 31681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:39.845063 31681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:39.847494 31681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:39.847671 31681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:39.851794 31681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34583
I20260810 21:13:39.851833 31797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34583 every 8 connection(s)
I20260810 21:13:39.852207 31681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:39.853193 31799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:39.855535 31799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2: Bootstrap starting.
I20260810 21:13:39.855608 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31681
I20260810 21:13:39.855717 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:39.855944 28117 external_mini_cluster.cc:1428] Setting key 8c0acc3276cc29088e93ec4cd518ea08
I20260810 21:13:39.856253 31799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:39.856616 31799 log.cc:824] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:39.857327 31799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2: No bootstrap required, opened a new log
I20260810 21:13:39.859112 31799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260810 21:13:39.859274 31799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:39.859333 31799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e328d06ab3241b3abd5561ce4aa30f2, State: Initialized, Role: FOLLOWER
I20260810 21:13:39.859477 31799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260810 21:13:39.859570 31799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:39.859614 31799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:39.859663 31799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:39.860188 31799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260810 21:13:39.860304 31799 leader_election.cc:302] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e328d06ab3241b3abd5561ce4aa30f2; no voters:
I20260810 21:13:39.860486 31799 leader_election.cc:288] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:39.860531 31813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:39.860694 31813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [term 1 LEADER]: Becoming Leader. State: Replica: 7e328d06ab3241b3abd5561ce4aa30f2, State: Running, Role: LEADER
I20260810 21:13:39.860747 31799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:39.860816 31813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260810 21:13:39.861151 31813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } } }
I20260810 21:13:39.861231 31813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:39.861416 31813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e328d06ab3241b3abd5561ce4aa30f2. Latest consensus state: current_term: 1 leader_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e328d06ab3241b3abd5561ce4aa30f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } } }
I20260810 21:13:39.861486 31813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:39.861583 31828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:39.862108 31828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 21:13:39 dist-test-slave-2kcd krb5kdc[31660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 21:13:39.863986 31828 catalog_manager.cc:1347] Generated new cluster ID: 28290f4b9e304aedb2078ac1c222bedb
I20260810 21:13:39.864038 31828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:39.867974 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:39.856484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35115 (local address 127.0.0.1:34583)
0810 21:13:39.856706 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:39.856737 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:39.856772 (+ 35us) server_negotiation.cc:407] Connection header received
0810 21:13:39.857248 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:39.857254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:39.857298 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:39.857444 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:39.858086 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:39.858971 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:39.859940 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:39.860388 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:39.860549 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:39.863621 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:39.863625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:39.863625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:39.863638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:39.866274 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:39.866745 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:39.866748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:39.866748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:39.866845 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:39.867119 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:39.867120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:39.867120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:39.867403 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:39.867550 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:39.867697 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 21:13:39.867790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260810 21:13:39.869026 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:39.873153 31828 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:39.873638 31828 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:39.877794 31806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:39.870611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56443 (local address 127.0.0.1:34583)
0810 21:13:39.870647 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:39.870649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:39.870801 (+ 152us) server_negotiation.cc:407] Connection header received
0810 21:13:39.870846 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:39.870847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:39.870857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:39.870915 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:39.871505 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:39.872325 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:39.873294 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:39.873605 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:39.873679 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:39.874916 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:39.874917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:39.874918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:39.874924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:39.876599 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:39.877000 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:39.877001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:39.877001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:39.877061 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:39.877287 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:39.877288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:39.877288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:39.877421 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:39.877525 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:39.877643 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 21:13:39.877706 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":18}
I20260810 21:13:39.883538 31828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e328d06ab3241b3abd5561ce4aa30f2: Generated new TSK 0
I20260810 21:13:39.883709 31828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:39.888127 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31681
W20260810 21:13:39.895967 31653 connection.cc:535] client connection to 127.0.0.1:34583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34583): Connection reset by peer (error 104)
2026-08-10T21:13:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:39 dist-test-slave-2kcd krb5kdc[31861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:39 dist-test-slave-2kcd krb5kdc[31861](info): set up 2 sockets
Aug 10 21:13:39 dist-test-slave-2kcd krb5kdc[31861](info): commencing operation
krb5kdc: starting...
W20260810 21:13:41.162333 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:41 dist-test-slave-2kcd krb5kdc[31861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:41.277484 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:41.358011 31885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:41.358201 31885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:41.358222 31885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:41.359261 31885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:41.359293 31885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:41.359308 31885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:41.359319 31885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:41.359331 31885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:41.360706 31885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:41.360942 31885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:41.361121 31885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:41.365787 31892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:41.365784 31894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:41.365790 31891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:41.366132 31885 server_base.cc:1034] running on GCE node
I20260810 21:13:41.366345 31885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:41.366611 31885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:41.367784 31885 hybrid_clock.cc:648] HybridClock initialized: now 1786396421367777 us; error 30 us; skew 500 ppm
Aug 10 21:13:41 dist-test-slave-2kcd krb5kdc[31861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:41.370253 31885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:41.370720 31885 webserver.cc:460] Webserver started at http://127.0.0.1:38125/ using document root <none> and password file <none>
I20260810 21:13:41.370945 31885 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:41.370991 31885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:41.371152 31885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:41.372217 31885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d"
format_stamp: "Formatted at 2026-08-10 21:13:41 on dist-test-slave-2kcd"
server_key: "a0eb8cfc7f327077cf44f03ac9d9c242"
server_key_iv: "e0825bd79805dde6f871c6d5f62800b5"
server_key_version: "encryptionkey@0"
I20260810 21:13:41.372556 31885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d"
format_stamp: "Formatted at 2026-08-10 21:13:41 on dist-test-slave-2kcd"
server_key: "a0eb8cfc7f327077cf44f03ac9d9c242"
server_key_iv: "e0825bd79805dde6f871c6d5f62800b5"
server_key_version: "encryptionkey@0"
I20260810 21:13:41.374022 31885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:41.374704 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:41.374890 31885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:13:41.374990 31885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d"
format_stamp: "Formatted at 2026-08-10 21:13:41 on dist-test-slave-2kcd"
server_key: "a0eb8cfc7f327077cf44f03ac9d9c242"
server_key_iv: "e0825bd79805dde6f871c6d5f62800b5"
server_key_version: "encryptionkey@0"
I20260810 21:13:41.375043 31885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:41.395371 31885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:41.397601 31885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:41.397709 31885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:41.401568 31885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41031
I20260810 21:13:41.401620 31999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41031 every 8 connection(s)
I20260810 21:13:41.401957 31885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:41.402158 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 31885
I20260810 21:13:41.402304 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:41.402535 28117 external_mini_cluster.cc:1428] Setting key 8ac1a6d655185a5de56eda10e3f3e868
I20260810 21:13:41.402980 32001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:41.406242 32001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d: Bootstrap starting.
I20260810 21:13:41.406922 32001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:41.407256 32001 log.cc:824] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:41.408054 32001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d: No bootstrap required, opened a new log
I20260810 21:13:41.408509 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:41.403056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52429 (local address 127.0.0.1:41031)
0810 21:13:41.403276 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:41.403285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:41.403315 (+ 30us) server_negotiation.cc:407] Connection header received
0810 21:13:41.404012 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:41.404018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:41.404065 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:41.404175 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:41.404433 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:41.405360 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:41.406444 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:41.407054 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:41.407170 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:41.407830 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:41.407833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:41.407833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:41.407987 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:41.408016 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:41.408026 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:41.408090 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:41.408269 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 21:13:41.408301 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":127,"threads_started":1}
I20260810 21:13:41.409621 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:41.410410 32001 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41031 } }
I20260810 21:13:41.410597 32001 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:41.410633 32001 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a788f3022c4f2e9f1af16e59fd9b2d, State: Initialized, Role: FOLLOWER
I20260810 21:13:41.410766 32001 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41031 } }
I20260810 21:13:41.410866 32001 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:41.410912 32001 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:41.410971 32001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:41.411480 32001 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41031 } }
I20260810 21:13:41.411559 32001 leader_election.cc:302] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3a788f3022c4f2e9f1af16e59fd9b2d; no voters:
I20260810 21:13:41.411679 32001 leader_election.cc:288] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:41.411754 32041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:41.411896 32041 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [term 1 LEADER]: Becoming Leader. State: Replica: e3a788f3022c4f2e9f1af16e59fd9b2d, State: Running, Role: LEADER
I20260810 21:13:41.411939 32001 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:41.411991 32041 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41031 } }
I20260810 21:13:41.412338 32042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41031 } } }
I20260810 21:13:41.412345 32045 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a788f3022c4f2e9f1af16e59fd9b2d. Latest consensus state: current_term: 1 leader_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a788f3022c4f2e9f1af16e59fd9b2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41031 } } }
I20260810 21:13:41.412504 32045 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:41.412480 32042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:41.412925 32053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:41.413410 32053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:41.414870 32053 catalog_manager.cc:1347] Generated new cluster ID: c7e5bab8c90e42b3ad54e94d8de493b6
I20260810 21:13:41.414927 32053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:41.416908 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:41.413483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54785 (local address 127.0.0.1:41031)
0810 21:13:41.413787 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:41.413789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:41.413797 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:13:41.413815 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:41.413815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:41.413823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:41.413872 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:41.414056 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:41.414674 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:41.415645 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:41.415942 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:41.416017 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:41.416619 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:41.416620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:41.416621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:41.416625 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:41.416638 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:41.416650 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:41.416714 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:41.416829 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 21:13:41.416839 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286}
I20260810 21:13:41.447516 32053 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:41.448083 32053 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:41.452840 32053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3a788f3022c4f2e9f1af16e59fd9b2d: Generated new TSK 0
I20260810 21:13:41.453032 32053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:41.462832 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 31885
W20260810 21:13:41.477391 31858 connection.cc:535] client connection to 127.0.0.1:41031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41031): Connection reset by peer (error 104)
2026-08-10T21:13:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:41 dist-test-slave-2kcd krb5kdc[32080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:41 dist-test-slave-2kcd krb5kdc[32080](info): set up 2 sockets
Aug 10 21:13:41 dist-test-slave-2kcd krb5kdc[32080](info): commencing operation
krb5kdc: starting...
W20260810 21:13:42.735641 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:42 dist-test-slave-2kcd krb5kdc[32080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:42.848935 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:42.928184 32102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:42.928369 32102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:42.928391 32102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:42.929519 32102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:42.929561 32102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:42.929575 32102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:42.929620 32102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:42.929644 32102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:42.931041 32102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:42.931257 32102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:42.931456 32102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:13:42.936151 32102 server_base.cc:1034] running on GCE node
W20260810 21:13:42.936198 32110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:42.936144 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:42.936144 32111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:42.936657 32102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:42.936942 32102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:42.938105 32102 hybrid_clock.cc:648] HybridClock initialized: now 1786396422938076 us; error 41 us; skew 500 ppm
Aug 10 21:13:42 dist-test-slave-2kcd krb5kdc[32080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:42.944470 32102 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:42.944959 32102 webserver.cc:460] Webserver started at http://127.0.0.1:45117/ using document root <none> and password file <none>
I20260810 21:13:42.945184 32102 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:42.945246 32102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:42.945377 32102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:42.946352 32102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "46216049001f4ca98da9f38bbceecdb6"
format_stamp: "Formatted at 2026-08-10 21:13:42 on dist-test-slave-2kcd"
server_key: "99d2a94588909203bc799e5e4cd2d16c"
server_key_iv: "8cdb72fbb022ba230281ca33efd9fc9e"
server_key_version: "encryptionkey@0"
I20260810 21:13:42.946698 32102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "46216049001f4ca98da9f38bbceecdb6"
format_stamp: "Formatted at 2026-08-10 21:13:42 on dist-test-slave-2kcd"
server_key: "99d2a94588909203bc799e5e4cd2d16c"
server_key_iv: "8cdb72fbb022ba230281ca33efd9fc9e"
server_key_version: "encryptionkey@0"
I20260810 21:13:42.948001 32102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:42.948699 32125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:42.948881 32102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:42.949015 32102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "46216049001f4ca98da9f38bbceecdb6"
format_stamp: "Formatted at 2026-08-10 21:13:42 on dist-test-slave-2kcd"
server_key: "99d2a94588909203bc799e5e4cd2d16c"
server_key_iv: "8cdb72fbb022ba230281ca33efd9fc9e"
server_key_version: "encryptionkey@0"
I20260810 21:13:42.949091 32102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:42.952214 32102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:42.953990 32102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:42.954139 32102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:42.958113 32102 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38465
I20260810 21:13:42.958144 32218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38465 every 8 connection(s)
I20260810 21:13:42.958442 32102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:42.959348 32220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:42.961537 32220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6: Bootstrap starting.
I20260810 21:13:42.962112 32220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:42.962395 32220 log.cc:824] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:42.963104 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 32102
I20260810 21:13:42.963138 32220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6: No bootstrap required, opened a new log
I20260810 21:13:42.963236 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:42.963446 28117 external_mini_cluster.cc:1428] Setting key b3f8836fa2bab8299653b47466f8fb46
I20260810 21:13:42.965638 32220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46216049001f4ca98da9f38bbceecdb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260810 21:13:42.965850 32220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:42.965895 32220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46216049001f4ca98da9f38bbceecdb6, State: Initialized, Role: FOLLOWER
I20260810 21:13:42.966078 32220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46216049001f4ca98da9f38bbceecdb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260810 21:13:42.966202 32220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:42.966246 32220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:42.966298 32220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:42.966926 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:42.963924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47675 (local address 127.0.0.1:38465)
0810 21:13:42.964102 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:42.964110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:42.964156 (+ 46us) server_negotiation.cc:407] Connection header received
0810 21:13:42.965582 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:42.965589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:42.965616 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:42.965725 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:42.966482 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:42.966486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:42.966487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:42.966491 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:42.966531 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:42.966545 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:42.966613 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:42.966674 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 21:13:42.966713 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260810 21:13:42.967049 32220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46216049001f4ca98da9f38bbceecdb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260810 21:13:42.967140 32220 leader_election.cc:302] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46216049001f4ca98da9f38bbceecdb6; no voters:
I20260810 21:13:42.967295 32220 leader_election.cc:288] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:42.967365 32228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:42.967528 32228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [term 1 LEADER]: Becoming Leader. State: Replica: 46216049001f4ca98da9f38bbceecdb6, State: Running, Role: LEADER
I20260810 21:13:42.967559 32220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:42.967626 32228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46216049001f4ca98da9f38bbceecdb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } }
I20260810 21:13:42.967949 32228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46216049001f4ca98da9f38bbceecdb6. Latest consensus state: current_term: 1 leader_uuid: "46216049001f4ca98da9f38bbceecdb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46216049001f4ca98da9f38bbceecdb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } } }
I20260810 21:13:42.968026 32228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:42.968240 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:42.968230 32229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46216049001f4ca98da9f38bbceecdb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46216049001f4ca98da9f38bbceecdb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38465 } } }
I20260810 21:13:42.968302 32229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:42.968994 32238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:42.969507 32238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:42.971091 32238 catalog_manager.cc:1347] Generated new cluster ID: 19627d4440cf4a93b336955a7ab15688
I20260810 21:13:42.971158 32227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:42.969903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52919 (local address 127.0.0.1:38465)
0810 21:13:42.969937 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:42.969939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:42.970137 (+ 198us) server_negotiation.cc:407] Connection header received
0810 21:13:42.970235 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:42.970236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:42.970241 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:42.970311 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:42.970877 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:42.970878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:42.970879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:42.970884 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:42.970898 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:42.970902 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:42.970932 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:42.971035 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 21:13:42.971048 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:13:42.971158 32238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:42.974895 32238 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:42.975039 32238 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:42.980254 32238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46216049001f4ca98da9f38bbceecdb6: Generated new TSK 0
I20260810 21:13:42.980436 32238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:42.989218 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 32102
2026-08-10T21:13:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:43 dist-test-slave-2kcd krb5kdc[32272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:43 dist-test-slave-2kcd krb5kdc[32272](info): set up 2 sockets
Aug 10 21:13:43 dist-test-slave-2kcd krb5kdc[32272](info): commencing operation
krb5kdc: starting...
W20260810 21:13:44.267192 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:44 dist-test-slave-2kcd krb5kdc[32272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:44.400312 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:44.480415 32297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:44.480612 32297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:44.480635 32297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:44.481813 32297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:44.481858 32297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:44.481873 32297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:44.481886 32297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:44.481936 32297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:44.483393 32297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:44.483631 32297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:44.483846 32297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:44.488540 32307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:44.488524 32305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:44.488765 32297 server_base.cc:1034] running on GCE node
W20260810 21:13:44.488782 32311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:44.489056 32297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:44.489341 32297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:44.490483 32297 hybrid_clock.cc:648] HybridClock initialized: now 1786396424490472 us; error 30 us; skew 500 ppm
Aug 10 21:13:44 dist-test-slave-2kcd krb5kdc[32272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:44.492941 32297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:44.493379 32297 webserver.cc:460] Webserver started at http://127.0.0.1:33199/ using document root <none> and password file <none>
I20260810 21:13:44.493558 32297 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:44.493598 32297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:44.493685 32297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:44.494630 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "c926de57f400463f85a0aee33151f7b3"
format_stamp: "Formatted at 2026-08-10 21:13:44 on dist-test-slave-2kcd"
server_key: "57a35bab276db24f14c9189353fd098a"
server_key_iv: "828ab464cb88dd3c639804da741508a4"
server_key_version: "encryptionkey@0"
I20260810 21:13:44.494925 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "c926de57f400463f85a0aee33151f7b3"
format_stamp: "Formatted at 2026-08-10 21:13:44 on dist-test-slave-2kcd"
server_key: "57a35bab276db24f14c9189353fd098a"
server_key_iv: "828ab464cb88dd3c639804da741508a4"
server_key_version: "encryptionkey@0"
I20260810 21:13:44.496157 32297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:13:44.496850 32325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:44.497018 32297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:44.497115 32297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "c926de57f400463f85a0aee33151f7b3"
format_stamp: "Formatted at 2026-08-10 21:13:44 on dist-test-slave-2kcd"
server_key: "57a35bab276db24f14c9189353fd098a"
server_key_iv: "828ab464cb88dd3c639804da741508a4"
server_key_version: "encryptionkey@0"
I20260810 21:13:44.497187 32297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:44.505546 32297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:44.507206 32297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:44.507352 32297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:44.511561 32297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33263
I20260810 21:13:44.511591 32423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33263 every 8 connection(s)
I20260810 21:13:44.511929 32297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:44.512717 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:44.514736 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 32297
I20260810 21:13:44.514865 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:44.515029 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3: Bootstrap starting.
I20260810 21:13:44.515062 28117 external_mini_cluster.cc:1428] Setting key 7d8971810d4798653ee332b979d723a0
I20260810 21:13:44.515726 32427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:44.516032 32427 log.cc:824] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:44.516667 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3: No bootstrap required, opened a new log
I20260810 21:13:44.518060 32427 raft_consensus.cc:348] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c926de57f400463f85a0aee33151f7b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260810 21:13:44.518205 32427 raft_consensus.cc:374] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:44.518253 32427 raft_consensus.cc:729] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c926de57f400463f85a0aee33151f7b3, State: Initialized, Role: FOLLOWER
I20260810 21:13:44.518375 32427 consensus_queue.cc:260] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c926de57f400463f85a0aee33151f7b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260810 21:13:44.518463 32427 raft_consensus.cc:388] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:44.518505 32427 raft_consensus.cc:482] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:44.518553 32427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:44.519085 32427 raft_consensus.cc:504] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c926de57f400463f85a0aee33151f7b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260810 21:13:44.519194 32427 leader_election.cc:302] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c926de57f400463f85a0aee33151f7b3; no voters:
I20260810 21:13:44.519356 32427 leader_election.cc:288] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:44.519385 32436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:44.519507 32436 raft_consensus.cc:686] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [term 1 LEADER]: Becoming Leader. State: Replica: c926de57f400463f85a0aee33151f7b3, State: Running, Role: LEADER
I20260810 21:13:44.519619 32427 sys_catalog.cc:564] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:44.519615 32436 consensus_queue.cc:237] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c926de57f400463f85a0aee33151f7b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } }
I20260810 21:13:44.520076 32436 sys_catalog.cc:455] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c926de57f400463f85a0aee33151f7b3. Latest consensus state: current_term: 1 leader_uuid: "c926de57f400463f85a0aee33151f7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c926de57f400463f85a0aee33151f7b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } } }
I20260810 21:13:44.520150 32436 sys_catalog.cc:458] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:44.520309 32441 sys_catalog.cc:455] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c926de57f400463f85a0aee33151f7b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c926de57f400463f85a0aee33151f7b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33263 } } }
I20260810 21:13:44.520385 32441 sys_catalog.cc:458] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:44.520906 32449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:44.521391 32449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:44.522882 32449 catalog_manager.cc:1347] Generated new cluster ID: 55c9b5b2e3b0403685fc7104c9647c94
Aug 10 21:13:44 dist-test-slave-2kcd krb5kdc[32272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 21:13:44.522940 32449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:44.526885 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:44.515558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46883 (local address 127.0.0.1:33263)
0810 21:13:44.515734 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:44.515741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:44.515765 (+ 24us) server_negotiation.cc:407] Connection header received
0810 21:13:44.516283 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:44.516290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:44.516335 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:44.516479 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:44.517190 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:44.518782 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:44.519765 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:44.520343 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:44.520452 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:44.523425 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:44.523429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:44.523429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:44.523439 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:44.525331 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:44.525750 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:44.525752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:44.525753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:44.525849 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:44.526057 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:44.526058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:44.526058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:44.526318 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:44.526472 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:44.526586 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 21:13:44.526705 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260810 21:13:44.528287 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:44.532401 32449 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:44.533097 32449 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:44.536067 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:44.529728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59239 (local address 127.0.0.1:33263)
0810 21:13:44.529759 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:44.529761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:44.529874 (+ 113us) server_negotiation.cc:407] Connection header received
0810 21:13:44.529935 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:44.529936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:44.529944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:44.529995 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:44.530612 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:44.531288 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:44.532224 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:44.532556 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:44.532634 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:44.533745 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:44.533746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:44.533746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:44.533753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:44.535019 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:44.535357 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:44.535358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:44.535359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:44.535415 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:44.535613 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:44.535614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:44.535614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:44.535714 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:44.535807 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:44.535903 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 21:13:44.535978 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:13:44.539078 32449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c926de57f400463f85a0aee33151f7b3: Generated new TSK 0
I20260810 21:13:44.539244 32449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:44.546782 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 32297
2026-08-10T21:13:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:44 dist-test-slave-2kcd krb5kdc[32490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:44 dist-test-slave-2kcd krb5kdc[32490](info): set up 2 sockets
Aug 10 21:13:44 dist-test-slave-2kcd krb5kdc[32490](info): commencing operation
krb5kdc: starting...
W20260810 21:13:45.822268 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:45 dist-test-slave-2kcd krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:45.937845 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:46.017490 32514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:46.017728 32514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:46.017765 32514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:46.018918 32514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:46.018977 32514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:46.019001 32514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:46.019021 32514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:46.019047 32514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:46.020488 32514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:46.020745 32514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:46.020992 32514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:46.025660 32522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:46.025671 32523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:46.025755 32525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:46.025923 32514 server_base.cc:1034] running on GCE node
I20260810 21:13:46.026403 32514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:46.026683 32514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:46.027850 32514 hybrid_clock.cc:648] HybridClock initialized: now 1786396426027815 us; error 29 us; skew 500 ppm
Aug 10 21:13:46 dist-test-slave-2kcd krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:46.030321 32514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:46.030717 32514 webserver.cc:460] Webserver started at http://127.0.0.1:41431/ using document root <none> and password file <none>
I20260810 21:13:46.030951 32514 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:46.031018 32514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:46.031157 32514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:46.032145 32514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "f377503f7066479a9a934fffc4dc5111"
format_stamp: "Formatted at 2026-08-10 21:13:46 on dist-test-slave-2kcd"
server_key: "565c8d7ceb0dfae7f0479d0d00638d7b"
server_key_iv: "4eaf66fd2f724057b869ecf26b64d734"
server_key_version: "encryptionkey@0"
I20260810 21:13:46.032486 32514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "f377503f7066479a9a934fffc4dc5111"
format_stamp: "Formatted at 2026-08-10 21:13:46 on dist-test-slave-2kcd"
server_key: "565c8d7ceb0dfae7f0479d0d00638d7b"
server_key_iv: "4eaf66fd2f724057b869ecf26b64d734"
server_key_version: "encryptionkey@0"
I20260810 21:13:46.033814 32514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:46.034546 32534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:46.034704 32514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:46.034801 32514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "f377503f7066479a9a934fffc4dc5111"
format_stamp: "Formatted at 2026-08-10 21:13:46 on dist-test-slave-2kcd"
server_key: "565c8d7ceb0dfae7f0479d0d00638d7b"
server_key_iv: "4eaf66fd2f724057b869ecf26b64d734"
server_key_version: "encryptionkey@0"
I20260810 21:13:46.034891 32514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:46.044533 32514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:46.046204 32514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:46.046348 32514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:46.050369 32514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46799
I20260810 21:13:46.050414 32619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46799 every 8 connection(s)
I20260810 21:13:46.050738 32514 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:46.051566 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:46.051990 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 32514
I20260810 21:13:46.052114 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:46.052277 28117 external_mini_cluster.cc:1428] Setting key 7c76a756c127d0cdda6db7272a49a751
I20260810 21:13:46.054523 32621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111: Bootstrap starting.
I20260810 21:13:46.055202 32621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:46.055512 32621 log.cc:824] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:46.056209 32621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111: No bootstrap required, opened a new log
I20260810 21:13:46.058274 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:46.052801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33337 (local address 127.0.0.1:46799)
0810 21:13:46.052980 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:46.052988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:46.053021 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:13:46.053523 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:46.053531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:46.053587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:46.053738 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:46.054032 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:46.055353 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:46.056161 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:46.056753 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:46.056927 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:46.057551 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:46.057555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:46.057556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:46.057757 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:46.057796 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:46.057812 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:46.057882 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:46.057953 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 21:13:46.058026 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260810 21:13:46.058569 32621 raft_consensus.cc:348] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f377503f7066479a9a934fffc4dc5111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46799 } }
I20260810 21:13:46.058714 32621 raft_consensus.cc:374] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:46.058759 32621 raft_consensus.cc:729] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f377503f7066479a9a934fffc4dc5111, State: Initialized, Role: FOLLOWER
I20260810 21:13:46.058876 32621 consensus_queue.cc:260] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f377503f7066479a9a934fffc4dc5111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46799 } }
I20260810 21:13:46.058981 32621 raft_consensus.cc:388] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:46.059018 32621 raft_consensus.cc:482] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:46.059057 32621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:46.059427 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:46.059729 32621 raft_consensus.cc:504] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f377503f7066479a9a934fffc4dc5111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46799 } }
I20260810 21:13:46.059834 32621 leader_election.cc:302] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f377503f7066479a9a934fffc4dc5111; no voters:
I20260810 21:13:46.060014 32621 leader_election.cc:288] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:46.060071 32627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:46.060230 32627 raft_consensus.cc:686] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [term 1 LEADER]: Becoming Leader. State: Replica: f377503f7066479a9a934fffc4dc5111, State: Running, Role: LEADER
I20260810 21:13:46.060257 32621 sys_catalog.cc:564] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:46.060365 32627 consensus_queue.cc:237] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f377503f7066479a9a934fffc4dc5111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46799 } }
I20260810 21:13:46.060727 32628 sys_catalog.cc:455] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f377503f7066479a9a934fffc4dc5111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f377503f7066479a9a934fffc4dc5111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46799 } } }
I20260810 21:13:46.060837 32628 sys_catalog.cc:458] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:46.061020 32628 sys_catalog.cc:455] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f377503f7066479a9a934fffc4dc5111. Latest consensus state: current_term: 1 leader_uuid: "f377503f7066479a9a934fffc4dc5111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f377503f7066479a9a934fffc4dc5111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46799 } } }
I20260810 21:13:46.061082 32628 sys_catalog.cc:458] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111 [sys.catalog]: This master's current role is: LEADER
W20260810 21:13:46.062186 32661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:13:46.062532 32661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:13:46.062633 32646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:46.062830 32646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:46.064324 32646 catalog_manager.cc:1347] Generated new cluster ID: 6e7b8199de8a408ea92abebe452d1e05
I20260810 21:13:46.064424 32646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:46.064476 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:46.061230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45087 (local address 127.0.0.1:46799)
0810 21:13:46.061267 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:46.061269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:46.061307 (+ 38us) server_negotiation.cc:407] Connection header received
0810 21:13:46.061326 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:46.061326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:46.061335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:46.061393 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:46.061597 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:46.062199 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:46.063183 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:46.063464 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:46.063552 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:46.064191 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:46.064192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:46.064192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:46.064196 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:46.064209 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:46.064214 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:46.064261 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:46.064342 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 21:13:46.064361 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:13:46.085456 32646 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:46.086231 32646 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:46.093169 32646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f377503f7066479a9a934fffc4dc5111: Generated new TSK 0
I20260810 21:13:46.093349 32646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:46.100283 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 32514
2026-08-10T21:13:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1548 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:46 dist-test-slave-2kcd krb5kdc[32671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:46 dist-test-slave-2kcd krb5kdc[32671](info): set up 2 sockets
Aug 10 21:13:46 dist-test-slave-2kcd krb5kdc[32671](info): commencing operation
krb5kdc: starting...
W20260810 21:13:47.365054 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:47 dist-test-slave-2kcd krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:47.479097 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:47.559396 32699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:47.559585 32699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:47.559607 32699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:47.560643 32699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:47.560675 32699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:47.560688 32699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:47.560700 32699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:47.560711 32699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:47.562103 32699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:47.562275 32699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:47.562433 32699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:47.567022 32712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:47.567022 32704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:47.567036 32705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:47.567242 32699 server_base.cc:1034] running on GCE node
I20260810 21:13:47.567476 32699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:47.567802 32699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:47.568953 32699 hybrid_clock.cc:648] HybridClock initialized: now 1786396427568938 us; error 28 us; skew 500 ppm
Aug 10 21:13:47 dist-test-slave-2kcd krb5kdc[32671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:47.571341 32699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:47.571713 32699 webserver.cc:460] Webserver started at http://127.0.0.1:41101/ using document root <none> and password file <none>
I20260810 21:13:47.571961 32699 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:47.572008 32699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:47.572156 32699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:47.573225 32699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "094884c89f764df9a98b30accafc4b40"
format_stamp: "Formatted at 2026-08-10 21:13:47 on dist-test-slave-2kcd"
server_key: "fc7123d961bc6bfee6abec67c7f39638"
server_key_iv: "bd6773343c34a4d7e85b6a4974ec215d"
server_key_version: "encryptionkey@0"
I20260810 21:13:47.573567 32699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "094884c89f764df9a98b30accafc4b40"
format_stamp: "Formatted at 2026-08-10 21:13:47 on dist-test-slave-2kcd"
server_key: "fc7123d961bc6bfee6abec67c7f39638"
server_key_iv: "bd6773343c34a4d7e85b6a4974ec215d"
server_key_version: "encryptionkey@0"
I20260810 21:13:47.574818 32699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:13:47.575482 32723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:47.575659 32699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:47.575752 32699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "094884c89f764df9a98b30accafc4b40"
format_stamp: "Formatted at 2026-08-10 21:13:47 on dist-test-slave-2kcd"
server_key: "fc7123d961bc6bfee6abec67c7f39638"
server_key_iv: "bd6773343c34a4d7e85b6a4974ec215d"
server_key_version: "encryptionkey@0"
I20260810 21:13:47.575814 32699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:47.579082 32699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:47.580844 32699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:47.580989 32699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:47.584964 32699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35785
I20260810 21:13:47.584995 331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35785 every 8 connection(s)
I20260810 21:13:47.585331 32699 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:47.586192 332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:47.588487 332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40: Bootstrap starting.
I20260810 21:13:47.589145 332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:47.589430 332 log.cc:824] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:47.590103 332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40: No bootstrap required, opened a new log
I20260810 21:13:47.591922 332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094884c89f764df9a98b30accafc4b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260810 21:13:47.592063 332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:47.592114 332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094884c89f764df9a98b30accafc4b40, State: Initialized, Role: FOLLOWER
I20260810 21:13:47.592296 332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094884c89f764df9a98b30accafc4b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260810 21:13:47.592381 332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:47.592432 332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:47.592494 332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:47.593132 332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094884c89f764df9a98b30accafc4b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260810 21:13:47.593241 332 leader_election.cc:302] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094884c89f764df9a98b30accafc4b40; no voters:
I20260810 21:13:47.593423 332 leader_election.cc:288] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:47.593463 335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:47.593637 335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [term 1 LEADER]: Becoming Leader. State: Replica: 094884c89f764df9a98b30accafc4b40, State: Running, Role: LEADER
I20260810 21:13:47.593678 332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:47.593906 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 32699
I20260810 21:13:47.594030 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:47.594209 28117 external_mini_cluster.cc:1428] Setting key d65b09f34b9641d4cc81c64dedd9bc12
I20260810 21:13:47.594347 335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094884c89f764df9a98b30accafc4b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } }
I20260810 21:13:47.595049 339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094884c89f764df9a98b30accafc4b40. Latest consensus state: current_term: 1 leader_uuid: "094884c89f764df9a98b30accafc4b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094884c89f764df9a98b30accafc4b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } } }
I20260810 21:13:47.595059 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094884c89f764df9a98b30accafc4b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094884c89f764df9a98b30accafc4b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35785 } } }
I20260810 21:13:47.595400 339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:47.595413 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:47.595654 363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:47.596190 363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:47.596765 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:47.594858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47831 (local address 127.0.0.1:35785)
0810 21:13:47.595067 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:47.595076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:47.595112 (+ 36us) server_negotiation.cc:407] Connection header received
0810 21:13:47.595286 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:47.595294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:47.595326 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:47.595737 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:47.596227 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:47.596230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:47.596230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:47.596234 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:47.596273 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:47.596287 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:47.596355 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:47.596446 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 21:13:47.596587 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260810 21:13:47.597790 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:47.597944 363 catalog_manager.cc:1347] Generated new cluster ID: ad4e98caa5404cb4b78541b7a6c3d2fc
I20260810 21:13:47.598002 363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:47.602205 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:47.601106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41859 (local address 127.0.0.1:35785)
0810 21:13:47.601146 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:47.601148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:47.601281 (+ 133us) server_negotiation.cc:407] Connection header received
0810 21:13:47.601344 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:47.601345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:47.601350 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:47.601406 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:47.602007 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:47.602008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:47.602009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:47.602013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:47.602026 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:47.602030 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:47.602079 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:47.602131 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 21:13:47.602142 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 21:13:47.612176 363 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:47.612340 363 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:47.615180 363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 094884c89f764df9a98b30accafc4b40: Generated new TSK 0
I20260810 21:13:47.615357 363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:47.621299 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 32699
2026-08-10T21:13:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:47 dist-test-slave-2kcd krb5kdc[380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:47 dist-test-slave-2kcd krb5kdc[380](info): set up 2 sockets
Aug 10 21:13:47 dist-test-slave-2kcd krb5kdc[380](info): commencing operation
krb5kdc: starting...
W20260810 21:13:48.900547 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:48 dist-test-slave-2kcd krb5kdc[380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:49.017537 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:49.097966 404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:49.098146 404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:49.098168 404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:49.099275 404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:49.099335 404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:49.099349 404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:49.099395 404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:49.099413 404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:49.100860 404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:49.101075 404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:49.101272 404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:49.106384 412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:49.106415 410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:49.106392 415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:49.106606 404 server_base.cc:1034] running on GCE node
I20260810 21:13:49.106981 404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:49.107286 404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:49.108446 404 hybrid_clock.cc:648] HybridClock initialized: now 1786396429108436 us; error 38 us; skew 500 ppm
Aug 10 21:13:49 dist-test-slave-2kcd krb5kdc[380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:49.115211 404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:49.115684 404 webserver.cc:460] Webserver started at http://127.0.0.1:33897/ using document root <none> and password file <none>
I20260810 21:13:49.115962 404 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:49.116029 404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:49.116178 404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:49.117277 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "30fc70b32d234458b2f0f2bc7ec26ed3"
format_stamp: "Formatted at 2026-08-10 21:13:49 on dist-test-slave-2kcd"
server_key: "f6a8ea83bffad1ace5dfd78b8a5f7931"
server_key_iv: "7258bd9f67469e9e59d9669b3162294f"
server_key_version: "encryptionkey@0"
I20260810 21:13:49.117623 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "30fc70b32d234458b2f0f2bc7ec26ed3"
format_stamp: "Formatted at 2026-08-10 21:13:49 on dist-test-slave-2kcd"
server_key: "f6a8ea83bffad1ace5dfd78b8a5f7931"
server_key_iv: "7258bd9f67469e9e59d9669b3162294f"
server_key_version: "encryptionkey@0"
I20260810 21:13:49.118937 404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:49.119640 431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:49.119801 404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:49.119901 404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "30fc70b32d234458b2f0f2bc7ec26ed3"
format_stamp: "Formatted at 2026-08-10 21:13:49 on dist-test-slave-2kcd"
server_key: "f6a8ea83bffad1ace5dfd78b8a5f7931"
server_key_iv: "7258bd9f67469e9e59d9669b3162294f"
server_key_version: "encryptionkey@0"
I20260810 21:13:49.119969 404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:49.135465 404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:49.137753 404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:49.137925 404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:49.142220 404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37785
I20260810 21:13:49.142254 512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37785 every 8 connection(s)
I20260810 21:13:49.142620 404 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:49.143487 516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:49.145820 516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3: Bootstrap starting.
I20260810 21:13:49.146407 516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:49.146703 516 log.cc:824] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:49.147420 516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3: No bootstrap required, opened a new log
I20260810 21:13:49.149094 516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37785 } }
I20260810 21:13:49.149233 516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:49.149278 516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fc70b32d234458b2f0f2bc7ec26ed3, State: Initialized, Role: FOLLOWER
I20260810 21:13:49.149430 516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37785 } }
I20260810 21:13:49.149492 516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:49.149535 516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:49.149586 516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:49.150133 516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37785 } }
I20260810 21:13:49.150260 516 leader_election.cc:302] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30fc70b32d234458b2f0f2bc7ec26ed3; no voters:
I20260810 21:13:49.150437 516 leader_election.cc:288] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:49.150458 519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:49.150625 519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [term 1 LEADER]: Becoming Leader. State: Replica: 30fc70b32d234458b2f0f2bc7ec26ed3, State: Running, Role: LEADER
I20260810 21:13:49.150700 516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:49.150758 519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37785 } }
I20260810 21:13:49.151197 521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30fc70b32d234458b2f0f2bc7ec26ed3. Latest consensus state: current_term: 1 leader_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37785 } } }
I20260810 21:13:49.151304 521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:49.151325 520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30fc70b32d234458b2f0f2bc7ec26ed3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37785 } } }
I20260810 21:13:49.151407 520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:49.151585 535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:49.152094 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 404
I20260810 21:13:49.152108 535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:49.152244 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:49.152443 28117 external_mini_cluster.cc:1428] Setting key dc82c0a995d0fb86cff5fda1a075531b
I20260810 21:13:49.153887 535 catalog_manager.cc:1347] Generated new cluster ID: 391ae18ec22247418f7235cdd5e7615c
I20260810 21:13:49.153949 535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:49.162184 535 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:49.162897 535 catalog_manager.cc:1504] Loading token signing keys...
Aug 10 21:13:49 dist-test-slave-2kcd krb5kdc[380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 21:13:49.166257 535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30fc70b32d234458b2f0f2bc7ec26ed3: Generated new TSK 0
I20260810 21:13:49.166440 535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:49.168473 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:49.152986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:48547 (local address 127.0.0.1:37785)
0810 21:13:49.156944 (+ 3958us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:49.156953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:49.156986 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:13:49.157097 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:49.157104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:49.157158 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:49.157290 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:49.158074 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:49.160081 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:49.161245 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:49.161846 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:49.161981 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:49.164848 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:49.164853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:49.164853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:49.164864 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:49.166854 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:49.167217 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:49.167220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:49.167220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:49.167324 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:49.167618 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:49.167619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:49.167620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:49.167902 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:49.168048 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:49.168154 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 21:13:49.168255 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3910,"thread_start_us":84,"threads_started":1}
I20260810 21:13:49.169492 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:49.177215 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:49.170972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:41057 (local address 127.0.0.1:37785)
0810 21:13:49.171005 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:49.171007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:49.171114 (+ 107us) server_negotiation.cc:407] Connection header received
0810 21:13:49.171142 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:49.171143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:49.171152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:49.171216 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:49.171665 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:49.172443 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:49.173176 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:49.173538 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:49.173609 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:49.174681 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:49.174682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:49.174683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:49.174689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:49.176125 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:49.176464 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:49.176465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:49.176466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:49.176522 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:49.176724 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:49.176725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:49.176725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:49.176864 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:49.176998 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:49.177070 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 21:13:49.177131 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:13:49.178798 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 404
W20260810 21:13:49.190807 377 connection.cc:535] client connection to 127.0.0.1:37785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37785): Connection reset by peer (error 104)
2026-08-10T21:13:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:49 dist-test-slave-2kcd krb5kdc[564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:49 dist-test-slave-2kcd krb5kdc[564](info): set up 2 sockets
Aug 10 21:13:49 dist-test-slave-2kcd krb5kdc[564](info): commencing operation
krb5kdc: starting...
W20260810 21:13:50.452482 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:50 dist-test-slave-2kcd krb5kdc[564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:50.571729 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:50.652506 587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:50.652693 587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:50.652714 587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:50.653816 587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:50.653869 587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:50.653883 587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:50.653895 587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:50.653908 587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:50.655308 587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:50.655494 587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:50.655653 587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:50.660261 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:50.660256 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:50.660416 603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:50.660518 587 server_base.cc:1034] running on GCE node
I20260810 21:13:50.660753 587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:50.661104 587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:50.662259 587 hybrid_clock.cc:648] HybridClock initialized: now 1786396430662251 us; error 36 us; skew 500 ppm
Aug 10 21:13:50 dist-test-slave-2kcd krb5kdc[564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:50.664760 587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:50.665257 587 webserver.cc:460] Webserver started at http://127.0.0.1:33421/ using document root <none> and password file <none>
I20260810 21:13:50.665480 587 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:50.665552 587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:50.665681 587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:50.666669 587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "599bd3da17d94c5589121fbea3aa2080"
format_stamp: "Formatted at 2026-08-10 21:13:50 on dist-test-slave-2kcd"
server_key: "03a7a675ec761b9dfd9f21ddc4835fe8"
server_key_iv: "3f4018aa2c1c253c477bb3965f343f21"
server_key_version: "encryptionkey@0"
I20260810 21:13:50.667098 587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "599bd3da17d94c5589121fbea3aa2080"
format_stamp: "Formatted at 2026-08-10 21:13:50 on dist-test-slave-2kcd"
server_key: "03a7a675ec761b9dfd9f21ddc4835fe8"
server_key_iv: "3f4018aa2c1c253c477bb3965f343f21"
server_key_version: "encryptionkey@0"
I20260810 21:13:50.668429 587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:13:50.669195 619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:50.669361 587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:50.669466 587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "599bd3da17d94c5589121fbea3aa2080"
format_stamp: "Formatted at 2026-08-10 21:13:50 on dist-test-slave-2kcd"
server_key: "03a7a675ec761b9dfd9f21ddc4835fe8"
server_key_iv: "3f4018aa2c1c253c477bb3965f343f21"
server_key_version: "encryptionkey@0"
I20260810 21:13:50.669552 587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:50.687569 587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:50.689786 587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:50.689944 587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:50.694101 587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43425
I20260810 21:13:50.694136 705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43425 every 8 connection(s)
I20260810 21:13:50.694473 587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:50.695268 709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:50.696673 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 587
I20260810 21:13:50.696781 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:50.696970 28117 external_mini_cluster.cc:1428] Setting key 298d8c5fc65c31b7d7b50bf7eea975c2
I20260810 21:13:50.697791 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080: Bootstrap starting.
I20260810 21:13:50.698491 709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:50.698872 709 log.cc:824] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:50.699632 709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080: No bootstrap required, opened a new log
I20260810 21:13:50.701745 709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599bd3da17d94c5589121fbea3aa2080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260810 21:13:50.701889 709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:50.701932 709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599bd3da17d94c5589121fbea3aa2080, State: Initialized, Role: FOLLOWER
I20260810 21:13:50.702073 709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599bd3da17d94c5589121fbea3aa2080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260810 21:13:50.702169 709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:50.702199 709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:50.702250 709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:50.702845 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:50.697444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:60407 (local address 127.0.0.1:43425)
0810 21:13:50.697657 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:50.697666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:50.697697 (+ 31us) server_negotiation.cc:407] Connection header received
0810 21:13:50.698312 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:50.698320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:50.698381 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:50.698528 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:50.698767 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:50.700051 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:50.700815 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:50.701478 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:50.701588 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:50.702156 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:50.702171 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:50.702172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:50.702377 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:50.702405 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:50.702415 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:50.702476 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:50.702601 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 21:13:50.702641 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260810 21:13:50.702867 709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599bd3da17d94c5589121fbea3aa2080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260810 21:13:50.702972 709 leader_election.cc:302] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 599bd3da17d94c5589121fbea3aa2080; no voters:
I20260810 21:13:50.703135 709 leader_election.cc:288] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:50.703166 714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:50.703274 714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [term 1 LEADER]: Becoming Leader. State: Replica: 599bd3da17d94c5589121fbea3aa2080, State: Running, Role: LEADER
I20260810 21:13:50.703379 709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:50.703414 714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599bd3da17d94c5589121fbea3aa2080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } }
I20260810 21:13:50.703838 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:50.703855 717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "599bd3da17d94c5589121fbea3aa2080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599bd3da17d94c5589121fbea3aa2080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } } }
I20260810 21:13:50.703954 717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:50.704130 718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599bd3da17d94c5589121fbea3aa2080. Latest consensus state: current_term: 1 leader_uuid: "599bd3da17d94c5589121fbea3aa2080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599bd3da17d94c5589121fbea3aa2080" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43425 } } }
I20260810 21:13:50.704174 726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:50.704272 718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:50.704913 726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:50.706823 726 catalog_manager.cc:1347] Generated new cluster ID: 10b73dbc4be74122865420a1f004c2db
I20260810 21:13:50.706880 726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:50.708997 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:50.705613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:33541 (local address 127.0.0.1:43425)
0810 21:13:50.705644 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:50.705645 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:50.705833 (+ 188us) server_negotiation.cc:407] Connection header received
0810 21:13:50.705882 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:50.705883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:50.705893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:50.705936 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:50.706078 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:50.706678 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:50.707684 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:50.707998 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:50.708099 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:50.708737 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:50.708739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:50.708739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:50.708743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:50.708757 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:50.708761 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:50.708852 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:50.708913 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 21:13:50.708924 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:13:50.714181 726 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:50.714830 726 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:50.721225 726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599bd3da17d94c5589121fbea3aa2080: Generated new TSK 0
I20260810 21:13:50.721379 726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:50.725140 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 587
W20260810 21:13:50.756372 561 connection.cc:535] client connection to 127.0.0.1:43425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43425): Connection reset by peer (error 104)
2026-08-10T21:13:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1565 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:50 dist-test-slave-2kcd krb5kdc[755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:50 dist-test-slave-2kcd krb5kdc[755](info): set up 2 sockets
Aug 10 21:13:50 dist-test-slave-2kcd krb5kdc[755](info): commencing operation
krb5kdc: starting...
W20260810 21:13:51.999805 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:52 dist-test-slave-2kcd krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:52.116945 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:52.197039 777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:52.197275 777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:52.197309 777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:52.198457 777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:52.198530 777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:52.198555 777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:52.198587 777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:52.198632 777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:52.200112 777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:52.200394 777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:52.200623 777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:52.205914 792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:52.205929 793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:52.206050 777 server_base.cc:1034] running on GCE node
W20260810 21:13:52.205914 795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:52.206344 777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:52.206634 777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:52.207792 777 hybrid_clock.cc:648] HybridClock initialized: now 1786396432207770 us; error 36 us; skew 500 ppm
Aug 10 21:13:52 dist-test-slave-2kcd krb5kdc[755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:52.210393 777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:52.210827 777 webserver.cc:460] Webserver started at http://127.0.0.1:38007/ using document root <none> and password file <none>
I20260810 21:13:52.211088 777 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:52.211153 777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:52.211292 777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:52.212306 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "e9e6edd10d97417a9999110a4554ec7f"
format_stamp: "Formatted at 2026-08-10 21:13:52 on dist-test-slave-2kcd"
server_key: "8023f5302674292e68f5d503a2fb69b3"
server_key_iv: "daefdb207505bd13a3496586c329b4d5"
server_key_version: "encryptionkey@0"
I20260810 21:13:52.212661 777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "e9e6edd10d97417a9999110a4554ec7f"
format_stamp: "Formatted at 2026-08-10 21:13:52 on dist-test-slave-2kcd"
server_key: "8023f5302674292e68f5d503a2fb69b3"
server_key_iv: "daefdb207505bd13a3496586c329b4d5"
server_key_version: "encryptionkey@0"
I20260810 21:13:52.213960 777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:52.214663 805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:52.214834 777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 21:13:52.214923 777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "e9e6edd10d97417a9999110a4554ec7f"
format_stamp: "Formatted at 2026-08-10 21:13:52 on dist-test-slave-2kcd"
server_key: "8023f5302674292e68f5d503a2fb69b3"
server_key_iv: "daefdb207505bd13a3496586c329b4d5"
server_key_version: "encryptionkey@0"
I20260810 21:13:52.215009 777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:52.218353 777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:52.220578 777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:52.220731 777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:52.224644 777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41409
I20260810 21:13:52.224675 897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41409 every 8 connection(s)
I20260810 21:13:52.225069 777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:52.225929 898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:52.228152 898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f: Bootstrap starting.
I20260810 21:13:52.228708 898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:52.229040 898 log.cc:824] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:52.229686 898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f: No bootstrap required, opened a new log
I20260810 21:13:52.230901 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 777
I20260810 21:13:52.231034 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:52.231241 28117 external_mini_cluster.cc:1428] Setting key aa09df1a0c5e030442dfff2988d14399
I20260810 21:13:52.231709 898 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e6edd10d97417a9999110a4554ec7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260810 21:13:52.231992 898 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:52.232074 898 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e6edd10d97417a9999110a4554ec7f, State: Initialized, Role: FOLLOWER
I20260810 21:13:52.232215 898 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e6edd10d97417a9999110a4554ec7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260810 21:13:52.232307 898 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:52.232345 898 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:52.232445 898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:52.233402 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:52.231799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:44241 (local address 127.0.0.1:41409)
0810 21:13:52.231997 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:52.232005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:52.232030 (+ 25us) server_negotiation.cc:407] Connection header received
0810 21:13:52.232146 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:52.232153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:52.232186 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:52.232310 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:52.232942 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:52.232945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:52.232945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:52.232948 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:52.232978 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:52.232987 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:52.233037 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:52.233133 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 21:13:52.233167 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260810 21:13:52.233613 898 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e6edd10d97417a9999110a4554ec7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260810 21:13:52.233723 898 leader_election.cc:302] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9e6edd10d97417a9999110a4554ec7f; no voters:
I20260810 21:13:52.233880 898 leader_election.cc:288] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:52.233918 907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:52.234113 907 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [term 1 LEADER]: Becoming Leader. State: Replica: e9e6edd10d97417a9999110a4554ec7f, State: Running, Role: LEADER
I20260810 21:13:52.234133 898 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:52.234256 907 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e6edd10d97417a9999110a4554ec7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260810 21:13:52.234370 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:52.234635 907 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9e6edd10d97417a9999110a4554ec7f. Latest consensus state: current_term: 1 leader_uuid: "e9e6edd10d97417a9999110a4554ec7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e6edd10d97417a9999110a4554ec7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } } }
I20260810 21:13:52.234707 907 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:52.234860 908 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9e6edd10d97417a9999110a4554ec7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9e6edd10d97417a9999110a4554ec7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } } }
I20260810 21:13:52.234917 908 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f [sys.catalog]: This master's current role is: LEADER
W20260810 21:13:52.236071 941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:13:52.236130 941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:13:52.236183 918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:52.236323 918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:52.237973 918 catalog_manager.cc:1347] Generated new cluster ID: b47f1877d0df47dba3e4dc94ed0d44b2
I20260810 21:13:52.238027 918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:52.238277 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:52.237047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:46793 (local address 127.0.0.1:41409)
0810 21:13:52.237104 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:52.237106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:52.237321 (+ 215us) server_negotiation.cc:407] Connection header received
0810 21:13:52.237383 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:52.237384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:52.237389 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:52.237455 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:52.238054 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:52.238059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:52.238059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:52.238063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:52.238076 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:52.238080 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:52.238132 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:52.238184 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 21:13:52.238200 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 21:13:52.264349 918 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:52.264556 918 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:52.274520 918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9e6edd10d97417a9999110a4554ec7f: Generated new TSK 0
I20260810 21:13:52.274749 918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:52.276257 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 777
2026-08-10T21:13:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:52 dist-test-slave-2kcd krb5kdc[955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:52 dist-test-slave-2kcd krb5kdc[955](info): set up 2 sockets
Aug 10 21:13:52 dist-test-slave-2kcd krb5kdc[955](info): commencing operation
krb5kdc: starting...
W20260810 21:13:53.559388 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:53 dist-test-slave-2kcd krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:53.679526 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37779
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:53.762564 977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:53.762787 977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:53.762826 977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:53.763935 977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:53.764003 977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:53.764030 977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:53.764060 977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:53.764082 977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:53.765640 977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:53.765900 977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:53.766134 977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:53.771265 985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:53.771255 991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:53.771255 984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:53.771580 977 server_base.cc:1034] running on GCE node
I20260810 21:13:53.771816 977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:53.772131 977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:53.773293 977 hybrid_clock.cc:648] HybridClock initialized: now 1786396433773277 us; error 38 us; skew 500 ppm
Aug 10 21:13:53 dist-test-slave-2kcd krb5kdc[955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:53.775870 977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:53.776381 977 webserver.cc:460] Webserver started at http://127.0.0.1:43131/ using document root <none> and password file <none>
I20260810 21:13:53.776643 977 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:53.776718 977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:53.776947 977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:53.778048 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "2faf4204e39c4b64baa76074709e2fe9"
format_stamp: "Formatted at 2026-08-10 21:13:53 on dist-test-slave-2kcd"
server_key: "2ebcaa2ea99b5e5eac5bab711528c2f5"
server_key_iv: "81574a7d89729b02c45477eec32204fe"
server_key_version: "encryptionkey@0"
I20260810 21:13:53.778365 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "2faf4204e39c4b64baa76074709e2fe9"
format_stamp: "Formatted at 2026-08-10 21:13:53 on dist-test-slave-2kcd"
server_key: "2ebcaa2ea99b5e5eac5bab711528c2f5"
server_key_iv: "81574a7d89729b02c45477eec32204fe"
server_key_version: "encryptionkey@0"
I20260810 21:13:53.779663 977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 21:13:53.780411 1001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:53.780706 977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:13:53.780840 977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "2faf4204e39c4b64baa76074709e2fe9"
format_stamp: "Formatted at 2026-08-10 21:13:53 on dist-test-slave-2kcd"
server_key: "2ebcaa2ea99b5e5eac5bab711528c2f5"
server_key_iv: "81574a7d89729b02c45477eec32204fe"
server_key_version: "encryptionkey@0"
I20260810 21:13:53.780921 977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:53.806151 977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:53.808192 977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:53.808375 977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:53.812897 977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37779
I20260810 21:13:53.812950 1090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37779 every 8 connection(s)
I20260810 21:13:53.813344 977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:53.814224 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:53.815078 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 977
I20260810 21:13:53.815204 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:53.815384 28117 external_mini_cluster.cc:1428] Setting key 0496800483b174748671815b3f02e8df
I20260810 21:13:53.817600 1091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9: Bootstrap starting.
I20260810 21:13:53.818351 1091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:53.818683 1091 log.cc:824] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:53.819548 1091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9: No bootstrap required, opened a new log
I20260810 21:13:53.821705 1091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf4204e39c4b64baa76074709e2fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37779 } }
I20260810 21:13:53.821846 1091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:53.821869 1091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2faf4204e39c4b64baa76074709e2fe9, State: Initialized, Role: FOLLOWER
I20260810 21:13:53.822010 1091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf4204e39c4b64baa76074709e2fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37779 } }
I20260810 21:13:53.822082 1091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:53.822103 1091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:53.822170 1091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 21:13:53 dist-test-slave-2kcd krb5kdc[955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 21:13:53.822775 1091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf4204e39c4b64baa76074709e2fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37779 } }
I20260810 21:13:53.822979 1091 leader_election.cc:302] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2faf4204e39c4b64baa76074709e2fe9; no voters:
I20260810 21:13:53.823225 1091 leader_election.cc:288] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:53.823447 1101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:53.823498 1091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:53.823613 1101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 2faf4204e39c4b64baa76074709e2fe9, State: Running, Role: LEADER
I20260810 21:13:53.823709 1101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf4204e39c4b64baa76074709e2fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37779 } }
I20260810 21:13:53.824141 1101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2faf4204e39c4b64baa76074709e2fe9. Latest consensus state: current_term: 1 leader_uuid: "2faf4204e39c4b64baa76074709e2fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf4204e39c4b64baa76074709e2fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37779 } } }
I20260810 21:13:53.824206 1099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2faf4204e39c4b64baa76074709e2fe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2faf4204e39c4b64baa76074709e2fe9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37779 } } }
I20260810 21:13:53.824254 1101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:53.824268 1099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:53.824544 1111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:53.825114 1111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:53.827227 1111 catalog_manager.cc:1347] Generated new cluster ID: 7596c02e5a5b4c3498237b6958cd50bf
I20260810 21:13:53.827283 1111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:53.827996 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:53.815984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:46405 (local address 127.0.0.1:37779)
0810 21:13:53.816198 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:53.816208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:53.816241 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:13:53.816823 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:53.816832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:53.816899 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:53.817052 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:53.817637 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:53.819037 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:53.819900 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:53.820331 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:53.820467 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:53.823293 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:53.823297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:53.823297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:53.823310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:53.826018 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:53.826511 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:53.826514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:53.826515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:53.826626 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:53.826867 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:53.826868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:53.826868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:53.827289 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:53.827467 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:53.827720 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 21:13:53.827823 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260810 21:13:53.829010 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:53.843669 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:53.837249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:36413 (local address 127.0.0.1:37779)
0810 21:13:53.837316 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:53.837318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:53.837464 (+ 146us) server_negotiation.cc:407] Connection header received
0810 21:13:53.837575 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:53.837576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:53.837588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:53.837644 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:53.838199 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:53.838691 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:53.839391 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:53.839645 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:53.839725 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:53.841038 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:53.841040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:53.841040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:53.841047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:53.842464 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:53.842841 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:53.842843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:53.842843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:53.842904 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:53.843140 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:53.843141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:53.843141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:53.843253 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:53.843367 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:53.843489 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 21:13:53.843556 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 21:13:53.852617 1111 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:53.853359 1111 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:53.868602 1111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2faf4204e39c4b64baa76074709e2fe9: Generated new TSK 0
I20260810 21:13:53.868854 1111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:53.877133 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 977
W20260810 21:13:53.893890 953 connection.cc:535] client connection to 127.0.0.1:37779 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37779): Connection reset by peer (error 104)
2026-08-10T21:13:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:53 dist-test-slave-2kcd krb5kdc[1142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:53 dist-test-slave-2kcd krb5kdc[1142](info): set up 2 sockets
Aug 10 21:13:53 dist-test-slave-2kcd krb5kdc[1142](info): commencing operation
krb5kdc: starting...
W20260810 21:13:55.195034 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:55 dist-test-slave-2kcd krb5kdc[1142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:55.316735 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:55.397732 1164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:55.397930 1164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:55.397953 1164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:55.399101 1164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:55.399143 1164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:55.399158 1164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:55.399200 1164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:55.399227 1164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:55.400660 1164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:55.400923 1164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:55.401089 1164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:55.405757 1177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:55.405764 1176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:55.407785 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:55.408488 1164 server_base.cc:1034] running on GCE node
I20260810 21:13:55.408689 1164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:55.409010 1164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:55.410162 1164 hybrid_clock.cc:648] HybridClock initialized: now 1786396435410150 us; error 28 us; skew 500 ppm
Aug 10 21:13:55 dist-test-slave-2kcd krb5kdc[1142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:55.412454 1164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:55.412958 1164 webserver.cc:460] Webserver started at http://127.0.0.1:35133/ using document root <none> and password file <none>
I20260810 21:13:55.413134 1164 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:55.413172 1164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:55.413262 1164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:55.414191 1164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "783bfc8670ed45b1b86d40161bd3ae65"
format_stamp: "Formatted at 2026-08-10 21:13:55 on dist-test-slave-2kcd"
server_key: "5b64b4813a1d52c1d5150eef0fbd714f"
server_key_iv: "c88962a3c8a6d89875d8caefcaa003c6"
server_key_version: "encryptionkey@0"
I20260810 21:13:55.414477 1164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "783bfc8670ed45b1b86d40161bd3ae65"
format_stamp: "Formatted at 2026-08-10 21:13:55 on dist-test-slave-2kcd"
server_key: "5b64b4813a1d52c1d5150eef0fbd714f"
server_key_iv: "c88962a3c8a6d89875d8caefcaa003c6"
server_key_version: "encryptionkey@0"
I20260810 21:13:55.415697 1164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:13:55.416378 1198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:55.416551 1164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:55.416658 1164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "783bfc8670ed45b1b86d40161bd3ae65"
format_stamp: "Formatted at 2026-08-10 21:13:55 on dist-test-slave-2kcd"
server_key: "5b64b4813a1d52c1d5150eef0fbd714f"
server_key_iv: "c88962a3c8a6d89875d8caefcaa003c6"
server_key_version: "encryptionkey@0"
I20260810 21:13:55.416730 1164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:55.426295 1164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:55.428033 1164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:55.428195 1164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:55.432914 1164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34509
I20260810 21:13:55.432950 1282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34509 every 8 connection(s)
I20260810 21:13:55.433305 1164 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:55.434137 1284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:55.436345 1284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65: Bootstrap starting.
I20260810 21:13:55.436969 1284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:55.437209 1284 log.cc:824] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:55.437839 1284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65: No bootstrap required, opened a new log
I20260810 21:13:55.439353 1284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783bfc8670ed45b1b86d40161bd3ae65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34509 } }
I20260810 21:13:55.439486 1284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:55.439509 1284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783bfc8670ed45b1b86d40161bd3ae65, State: Initialized, Role: FOLLOWER
I20260810 21:13:55.439654 1284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783bfc8670ed45b1b86d40161bd3ae65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34509 } }
I20260810 21:13:55.439739 1284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:55.439762 1284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:55.439790 1284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:55.440251 1284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783bfc8670ed45b1b86d40161bd3ae65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34509 } }
I20260810 21:13:55.440335 1284 leader_election.cc:302] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 783bfc8670ed45b1b86d40161bd3ae65; no voters:
I20260810 21:13:55.440448 1284 leader_election.cc:288] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:55.440529 1288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:55.440698 1288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [term 1 LEADER]: Becoming Leader. State: Replica: 783bfc8670ed45b1b86d40161bd3ae65, State: Running, Role: LEADER
I20260810 21:13:55.440733 1284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:55.440809 1288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783bfc8670ed45b1b86d40161bd3ae65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34509 } }
I20260810 21:13:55.441282 1292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 783bfc8670ed45b1b86d40161bd3ae65. Latest consensus state: current_term: 1 leader_uuid: "783bfc8670ed45b1b86d40161bd3ae65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783bfc8670ed45b1b86d40161bd3ae65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34509 } } }
I20260810 21:13:55.441421 1292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:55.441706 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1164
I20260810 21:13:55.441846 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:55.441957 1302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:55.442070 28117 external_mini_cluster.cc:1428] Setting key 714e9eab103778ebff3f24c525975b65
I20260810 21:13:55.441401 1291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "783bfc8670ed45b1b86d40161bd3ae65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "783bfc8670ed45b1b86d40161bd3ae65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34509 } } }
I20260810 21:13:55.442394 1291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65 [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:55.442521 1302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:55.444161 1302 catalog_manager.cc:1347] Generated new cluster ID: 996448ffd5574c0897d55b2006d706f8
I20260810 21:13:55.444217 1302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:55.448350 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:55.442709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:56073 (local address 127.0.0.1:34509)
0810 21:13:55.443090 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:55.443100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:55.443129 (+ 29us) server_negotiation.cc:407] Connection header received
0810 21:13:55.443240 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:55.443246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:55.443301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:55.443435 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:55.443695 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:55.445115 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:55.446235 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:55.446823 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:55.446912 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:55.447659 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:55.447663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:55.447663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:55.447862 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:55.447899 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:55.447912 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:55.447990 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:55.448110 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 21:13:55.448134 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":77,"threads_started":1}
I20260810 21:13:55.449344 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:55.456586 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:55.453036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:40503 (local address 127.0.0.1:34509)
0810 21:13:55.453072 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:55.453074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:55.453251 (+ 177us) server_negotiation.cc:407] Connection header received
0810 21:13:55.453293 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:55.453294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:55.453305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:55.453363 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:55.453462 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:55.454222 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:55.455249 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:55.455609 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:55.455680 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:55.456334 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:55.456335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:55.456335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:55.456340 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:55.456354 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:55.456358 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:55.456421 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:55.456493 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 21:13:55.456505 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:13:55.465860 1302 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:55.466518 1302 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:55.472297 1302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 783bfc8670ed45b1b86d40161bd3ae65: Generated new TSK 0
I20260810 21:13:55.472483 1302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:55.477874 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1164
W20260810 21:13:55.494663 1140 connection.cc:535] client connection to 127.0.0.1:34509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34509): Connection reset by peer (error 104)
2026-08-10T21:13:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1600 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:55 dist-test-slave-2kcd krb5kdc[1335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:55 dist-test-slave-2kcd krb5kdc[1335](info): set up 2 sockets
Aug 10 21:13:55 dist-test-slave-2kcd krb5kdc[1335](info): commencing operation
krb5kdc: starting...
W20260810 21:13:56.759461 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:56 dist-test-slave-2kcd krb5kdc[1335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 21:13:56.881733 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:56.966984 1365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:56.967190 1365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:56.967213 1365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:56.968354 1365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:56.968401 1365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:56.968418 1365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:56.968477 1365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:56.968503 1365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:56.970090 1365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:56.970353 1365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:56.970599 1365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:56.975580 1370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:56.975586 1372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:56.975703 1365 server_base.cc:1034] running on GCE node
W20260810 21:13:56.975826 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:56.976055 1365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:56.976383 1365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:56.977551 1365 hybrid_clock.cc:648] HybridClock initialized: now 1786396436977541 us; error 42 us; skew 500 ppm
Aug 10 21:13:56 dist-test-slave-2kcd krb5kdc[1335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:56.979984 1365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 21:13:56.980479 1365 webserver.cc:460] Webserver started at http://127.0.0.1:39221/ using document root <none> and password file <none>
I20260810 21:13:56.980710 1365 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:56.980777 1365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:56.980973 1365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:56.981989 1365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "3ce38fb40614421f8eb26ac97d1874bd"
format_stamp: "Formatted at 2026-08-10 21:13:56 on dist-test-slave-2kcd"
server_key: "764c455a9e1390eb1803b0cd3a0bd13f"
server_key_iv: "a4fcb09906f24de30434b72e1574b747"
server_key_version: "encryptionkey@0"
I20260810 21:13:56.982347 1365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "3ce38fb40614421f8eb26ac97d1874bd"
format_stamp: "Formatted at 2026-08-10 21:13:56 on dist-test-slave-2kcd"
server_key: "764c455a9e1390eb1803b0cd3a0bd13f"
server_key_iv: "a4fcb09906f24de30434b72e1574b747"
server_key_version: "encryptionkey@0"
I20260810 21:13:56.983649 1365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:13:56.984390 1389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:56.984597 1365 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:56.984715 1365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "3ce38fb40614421f8eb26ac97d1874bd"
format_stamp: "Formatted at 2026-08-10 21:13:56 on dist-test-slave-2kcd"
server_key: "764c455a9e1390eb1803b0cd3a0bd13f"
server_key_iv: "a4fcb09906f24de30434b72e1574b747"
server_key_version: "encryptionkey@0"
I20260810 21:13:56.984844 1365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:56.988380 1365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:56.990433 1365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:56.990617 1365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:56.994954 1365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44597
I20260810 21:13:56.994982 1475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44597 every 8 connection(s)
I20260810 21:13:56.995313 1365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:56.996124 1477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:56.997097 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1365
I20260810 21:13:56.997228 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:56.997442 28117 external_mini_cluster.cc:1428] Setting key 5c666f70b439bac132299ae71021fb15
I20260810 21:13:56.998849 1477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd: Bootstrap starting.
I20260810 21:13:56.999526 1477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:56.999861 1477 log.cc:824] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:57.000020 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:56.997954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:59125 (local address 127.0.0.1:44597)
0810 21:13:56.998146 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:56.998155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:56.998187 (+ 32us) server_negotiation.cc:407] Connection header received
0810 21:13:56.998697 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:56.998706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:56.998737 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:56.998902 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:56.999576 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:56.999579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:56.999580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:56.999584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:56.999626 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:56.999643 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:56.999748 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:56.999781 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 21:13:56.999822 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260810 21:13:57.000684 1477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd: No bootstrap required, opened a new log
I20260810 21:13:57.000917 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:57.002681 1477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce38fb40614421f8eb26ac97d1874bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260810 21:13:57.002861 1477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:57.002913 1477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce38fb40614421f8eb26ac97d1874bd, State: Initialized, Role: FOLLOWER
I20260810 21:13:57.003034 1477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce38fb40614421f8eb26ac97d1874bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260810 21:13:57.003150 1477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:57.003188 1477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:57.003240 1477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:57.003557 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:57.002376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.94:46243 (local address 127.0.0.1:44597)
0810 21:13:57.002443 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:57.002446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:57.002586 (+ 140us) server_negotiation.cc:407] Connection header received
0810 21:13:57.002628 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:57.002629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:57.002633 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:57.002694 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:57.003286 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:57.003287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:57.003288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:13:57.003294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:57.003317 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:13:57.003322 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:57.003396 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:57.003467 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 21:13:57.003478 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 21:13:57.004000 1477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce38fb40614421f8eb26ac97d1874bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260810 21:13:57.004104 1477 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce38fb40614421f8eb26ac97d1874bd; no voters:
I20260810 21:13:57.004225 1477 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:57.004302 1497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:57.004402 1477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:57.004467 1497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [term 1 LEADER]: Becoming Leader. State: Replica: 3ce38fb40614421f8eb26ac97d1874bd, State: Running, Role: LEADER
I20260810 21:13:57.004562 1497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce38fb40614421f8eb26ac97d1874bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } }
I20260810 21:13:57.004973 1497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce38fb40614421f8eb26ac97d1874bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce38fb40614421f8eb26ac97d1874bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } } }
I20260810 21:13:57.005066 1497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:57.005257 1497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce38fb40614421f8eb26ac97d1874bd. Latest consensus state: current_term: 1 leader_uuid: "3ce38fb40614421f8eb26ac97d1874bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce38fb40614421f8eb26ac97d1874bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44597 } } }
I20260810 21:13:57.005347 1512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:57.005389 1497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd [sys.catalog]: This master's current role is: LEADER
I20260810 21:13:57.005919 1512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:57.007442 1512 catalog_manager.cc:1347] Generated new cluster ID: 0962e27a55ce4579b1b1db05e9fa0b52
I20260810 21:13:57.007493 1512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:57.020852 1512 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:57.021013 1512 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:57.025895 1512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ce38fb40614421f8eb26ac97d1874bd: Generated new TSK 0
I20260810 21:13:57.026078 1512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:57.031646 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1365
2026-08-10T21:13:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1551 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:57 dist-test-slave-2kcd krb5kdc[1535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:57 dist-test-slave-2kcd krb5kdc[1535](info): set up 2 sockets
Aug 10 21:13:57 dist-test-slave-2kcd krb5kdc[1535](info): commencing operation
krb5kdc: starting...
W20260810 21:13:58.321513 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:58 dist-test-slave-2kcd krb5kdc[1535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:13:58.439867 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35711
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:13:58.522395 1567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:13:58.522584 1567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:13:58.522606 1567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:13:58.523742 1567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:13:58.523785 1567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:13:58.523800 1567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:13:58.523813 1567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:13:58.523851 1567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:13:58.525378 1567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:35711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:13:58.525633 1567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:13:58.525861 1567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:13:58.530648 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:58.530668 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:13:58.530711 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:13:58.530820 1567 server_base.cc:1034] running on GCE node
I20260810 21:13:58.531162 1567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:13:58.531437 1567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:13:58.532608 1567 hybrid_clock.cc:648] HybridClock initialized: now 1786396438532597 us; error 31 us; skew 500 ppm
Aug 10 21:13:58 dist-test-slave-2kcd krb5kdc[1535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396438, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:13:58.535074 1567 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:13:58.535463 1567 webserver.cc:460] Webserver started at http://127.27.117.126:42767/ using document root <none> and password file <none>
I20260810 21:13:58.535655 1567 fs_manager.cc:359] Metadata directory not provided
I20260810 21:13:58.535694 1567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:13:58.535782 1567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:13:58.536717 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e"
format_stamp: "Formatted at 2026-08-10 21:13:58 on dist-test-slave-2kcd"
server_key: "d9457189815265b80209f305a93bb5a0"
server_key_iv: "a9fc4b8d6806392786c805d34c5acb33"
server_key_version: "encryptionkey@0"
I20260810 21:13:58.537081 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e"
format_stamp: "Formatted at 2026-08-10 21:13:58 on dist-test-slave-2kcd"
server_key: "d9457189815265b80209f305a93bb5a0"
server_key_iv: "a9fc4b8d6806392786c805d34c5acb33"
server_key_version: "encryptionkey@0"
I20260810 21:13:58.538296 1567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:13:58.538954 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:13:58.539111 1567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:13:58.539202 1567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e"
format_stamp: "Formatted at 2026-08-10 21:13:58 on dist-test-slave-2kcd"
server_key: "d9457189815265b80209f305a93bb5a0"
server_key_iv: "a9fc4b8d6806392786c805d34c5acb33"
server_key_version: "encryptionkey@0"
I20260810 21:13:58.539256 1567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:13:58.556135 1567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:13:58.557955 1567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:13:58.558072 1567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:13:58.562338 1567 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:35711
I20260810 21:13:58.562443 1677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:35711 every 8 connection(s)
I20260810 21:13:58.562734 1567 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:13:58.563601 1679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:13:58.564953 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1567
I20260810 21:13:58.565069 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:13:58.565263 28117 external_mini_cluster.cc:1428] Setting key f36f5ba3ab784f922823d92f83119f8a
I20260810 21:13:58.566156 1679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e: Bootstrap starting.
I20260810 21:13:58.566735 1679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e: Neither blocks nor log segments found. Creating new log.
I20260810 21:13:58.567022 1679 log.cc:824] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e: Log is configured to *not* fsync() on all Append() calls
I20260810 21:13:58.567785 1679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e: No bootstrap required, opened a new log
I20260810 21:13:58.570109 1679 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:13:58.570298 1679 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:13:58.570330 1679 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6cf8fa46af141c1a4a1ea8e6d65ed6e, State: Initialized, Role: FOLLOWER
I20260810 21:13:58.570453 1679 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:13:58.570549 1679 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:13:58.570580 1679 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:13:58.570672 1679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:13:58.571174 1679 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:13:58.571288 1679 leader_election.cc:302] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6cf8fa46af141c1a4a1ea8e6d65ed6e; no voters:
I20260810 21:13:58.571487 1679 leader_election.cc:288] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:13:58.571504 1696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:13:58.571622 1696 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [term 1 LEADER]: Becoming Leader. State: Replica: a6cf8fa46af141c1a4a1ea8e6d65ed6e, State: Running, Role: LEADER
I20260810 21:13:58.571746 1679 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:13:58.571754 1696 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } }
I20260810 21:13:58.572175 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6cf8fa46af141c1a4a1ea8e6d65ed6e. Latest consensus state: current_term: 1 leader_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } } }
I20260810 21:13:58.572254 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [sys.catalog]: This master's current role is: LEADER
Aug 10 21:13:58 dist-test-slave-2kcd krb5kdc[1535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:13:58.572530 1699 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6cf8fa46af141c1a4a1ea8e6d65ed6e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 35711 } } }
I20260810 21:13:58.572808 1699 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e [sys.catalog]: This master's current role is: LEADER
W20260810 21:13:58.573740 1725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:13:58.573797 1725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:13:58.573946 1705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:13:58.574126 1705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:13:58.575654 1705 catalog_manager.cc:1347] Generated new cluster ID: 190cff63c9e74f11bda8d35a8db0bfa3
I20260810 21:13:58.575703 1705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:13:58.577068 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:58.565776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41955 (local address 127.27.117.126:35711)
0810 21:13:58.565977 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:58.565984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:58.566016 (+ 32us) server_negotiation.cc:407] Connection header received
0810 21:13:58.566442 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:58.566448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:58.566495 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:58.566632 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:58.567387 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:58.568861 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:58.569764 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:58.570402 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:58.570512 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:58.573422 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:58.573426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:58.573426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:58.573439 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:58.575300 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:58.575758 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:58.575760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:58.575761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:58.575862 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:58.576137 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:58.576138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:58.576138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:58.576406 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:58.576579 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:58.576750 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 21:13:58.576876 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260810 21:13:58.585565 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:13:58.586779 1705 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:13:58.587466 1705 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:13:58.593380 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:13:58.587119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56441 (local address 127.27.117.126:35711)
0810 21:13:58.587155 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:13:58.587157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:13:58.587271 (+ 114us) server_negotiation.cc:407] Connection header received
0810 21:13:58.587335 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:13:58.587336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:13:58.587482 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:13:58.587554 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:13:58.588078 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:58.588583 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:58.589348 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:13:58.589571 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:13:58.589655 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:13:58.590823 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:13:58.590824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:13:58.590824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:13:58.590831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:13:58.592218 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:58.592582 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:58.592584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:58.592584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:58.592646 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:13:58.592877 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:13:58.592878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:13:58.592878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:13:58.592987 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:13:58.593094 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 21:13:58.593211 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 21:13:58.593273 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:13:58.595132 1705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6cf8fa46af141c1a4a1ea8e6d65ed6e: Generated new TSK 0
I20260810 21:13:58.595309 1705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:13:58.597685 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1567
W20260810 21:13:58.616205 1530 connection.cc:535] client connection to 127.27.117.126:35711 recv error: Network error: failed to read from TLS socket (remote: 127.27.117.126:35711): Connection reset by peer (error 104)
2026-08-10T21:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:13:58 dist-test-slave-2kcd krb5kdc[1744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:13:58 dist-test-slave-2kcd krb5kdc[1744](info): set up 2 sockets
Aug 10 21:13:58 dist-test-slave-2kcd krb5kdc[1744](info): commencing operation
krb5kdc: starting...
W20260810 21:13:59.878993 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:13:59 dist-test-slave-2kcd krb5kdc[1744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:13:59.999759 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45511
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:00.079952 1776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:00.080195 1776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:00.080238 1776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:00.081388 1776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:00.081477 1776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:00.081514 1776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:00.081557 1776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:00.081583 1776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:00.083060 1776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:45511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:00.083343 1776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:00.083597 1776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:00.088335 1791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:00.088399 1783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:00.088575 1782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:00.088531 1776 server_base.cc:1034] running on GCE node
I20260810 21:14:00.088882 1776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:00.089169 1776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:00.090317 1776 hybrid_clock.cc:648] HybridClock initialized: now 1786396440090302 us; error 29 us; skew 500 ppm
Aug 10 21:14:00 dist-test-slave-2kcd krb5kdc[1744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396440, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:00.098028 1776 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:00.101737 1776 webserver.cc:460] Webserver started at http://127.27.117.126:33189/ using document root <none> and password file <none>
I20260810 21:14:00.101984 1776 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:00.102052 1776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:00.102228 1776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:00.103708 1776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "06a945e4c40b4f3f8ed973000dcce62f"
format_stamp: "Formatted at 2026-08-10 21:14:00 on dist-test-slave-2kcd"
server_key: "0ec22eeee10950aa4b14be7ab77e88e1"
server_key_iv: "d749ea281689291c1a229b69ddf81e86"
server_key_version: "encryptionkey@0"
I20260810 21:14:00.104140 1776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "06a945e4c40b4f3f8ed973000dcce62f"
format_stamp: "Formatted at 2026-08-10 21:14:00 on dist-test-slave-2kcd"
server_key: "0ec22eeee10950aa4b14be7ab77e88e1"
server_key_iv: "d749ea281689291c1a229b69ddf81e86"
server_key_version: "encryptionkey@0"
I20260810 21:14:00.105568 1776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260810 21:14:00.106351 1800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:00.106564 1776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:14:00.106675 1776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "06a945e4c40b4f3f8ed973000dcce62f"
format_stamp: "Formatted at 2026-08-10 21:14:00 on dist-test-slave-2kcd"
server_key: "0ec22eeee10950aa4b14be7ab77e88e1"
server_key_iv: "d749ea281689291c1a229b69ddf81e86"
server_key_version: "encryptionkey@0"
I20260810 21:14:00.106756 1776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:00.129722 1776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:00.131975 1776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:00.132133 1776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:00.136267 1776 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:45511
I20260810 21:14:00.136303 1873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:45511 every 8 connection(s)
I20260810 21:14:00.136660 1776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:00.137560 1875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:00.139729 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f: Bootstrap starting.
I20260810 21:14:00.140256 1875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:00.140502 1875 log.cc:824] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:00.141242 1875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f: No bootstrap required, opened a new log
I20260810 21:14:00.142773 1875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a945e4c40b4f3f8ed973000dcce62f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45511 } }
I20260810 21:14:00.142901 1875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:14:00.142923 1875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a945e4c40b4f3f8ed973000dcce62f, State: Initialized, Role: FOLLOWER
I20260810 21:14:00.143054 1875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a945e4c40b4f3f8ed973000dcce62f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45511 } }
I20260810 21:14:00.143141 1875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:14:00.143162 1875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:14:00.143190 1875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:00.143957 1875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a945e4c40b4f3f8ed973000dcce62f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45511 } }
I20260810 21:14:00.144044 1875 leader_election.cc:302] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06a945e4c40b4f3f8ed973000dcce62f; no voters:
I20260810 21:14:00.144174 1875 leader_election.cc:288] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:14:00.144228 1878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:00.144411 1878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [term 1 LEADER]: Becoming Leader. State: Replica: 06a945e4c40b4f3f8ed973000dcce62f, State: Running, Role: LEADER
I20260810 21:14:00.144429 1875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:00.144550 1878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a945e4c40b4f3f8ed973000dcce62f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45511 } }
I20260810 21:14:00.144663 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1776
I20260810 21:14:00.144834 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:00.145025 28117 external_mini_cluster.cc:1428] Setting key 24e804c4cb237a80613e94509d54a2cb
I20260810 21:14:00.144999 1880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06a945e4c40b4f3f8ed973000dcce62f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a945e4c40b4f3f8ed973000dcce62f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45511 } } }
I20260810 21:14:00.145023 1881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06a945e4c40b4f3f8ed973000dcce62f. Latest consensus state: current_term: 1 leader_uuid: "06a945e4c40b4f3f8ed973000dcce62f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06a945e4c40b4f3f8ed973000dcce62f" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 45511 } } }
I20260810 21:14:00.145085 1880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:00.145092 1881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f [sys.catalog]: This master's current role is: LEADER
W20260810 21:14:00.146613 1909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:00.146690 1909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:14:00.146775 1889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:14:00.146946 1889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:00.148370 1889 catalog_manager.cc:1347] Generated new cluster ID: a1d2efd94aa44a36ac9e5837b81dcf7a
I20260810 21:14:00.148427 1889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:00.151347 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:00.145694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42475 (local address 127.27.117.126:45511)
0810 21:14:00.145880 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:00.145890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:00.145927 (+ 37us) server_negotiation.cc:407] Connection header received
0810 21:14:00.146065 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:00.146073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:00.146147 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:00.146356 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:00.146610 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:00.148004 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:00.149170 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:00.149787 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:00.149904 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:00.150635 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:00.150639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:00.150640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:00.150861 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:00.150900 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:00.150914 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:00.151016 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:00.151097 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 21:14:00.151125 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260810 21:14:00.152262 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:00.157224 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:00.153863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59595 (local address 127.27.117.126:45511)
0810 21:14:00.153907 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:00.153909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:00.153917 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:14:00.153977 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:00.153977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:00.153985 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:00.154039 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:00.154156 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:00.154869 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:00.155882 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:00.156215 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:00.156275 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:00.156949 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:00.156951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:00.156951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:00.156955 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:00.156969 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:00.156974 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:00.157032 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:00.157131 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 21:14:00.157144 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:14:00.159242 1889 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:00.159719 1889 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:00.164081 1889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06a945e4c40b4f3f8ed973000dcce62f: Generated new TSK 0
I20260810 21:14:00.164250 1889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:14:00.167688 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1776
W20260810 21:14:00.185051 1739 connection.cc:535] client connection to 127.27.117.126:45511 recv error: Network error: failed to read from TLS socket (remote: 127.27.117.126:45511): Connection reset by peer (error 104)
2026-08-10T21:14:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1568 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:14:00 dist-test-slave-2kcd krb5kdc[1927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:14:00 dist-test-slave-2kcd krb5kdc[1927](info): set up 2 sockets
Aug 10 21:14:00 dist-test-slave-2kcd krb5kdc[1927](info): commencing operation
krb5kdc: starting...
W20260810 21:14:01.452303 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:14:01 dist-test-slave-2kcd krb5kdc[1927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:14:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:14:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:14:01.571238 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:43565
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:01.650638 1952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:01.650868 1952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:01.650905 1952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:01.652014 1952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:01.652081 1952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:01.652104 1952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:01.652125 1952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:01.652154 1952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:01.653635 1952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:43565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:01.653899 1952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:01.654115 1952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:01.658841 1970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:01.658965 1952 server_base.cc:1034] running on GCE node
W20260810 21:14:01.658972 1968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:01.660240 1967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:01.660516 1952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:01.660840 1952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:01.662014 1952 hybrid_clock.cc:648] HybridClock initialized: now 1786396441662001 us; error 26 us; skew 500 ppm
Aug 10 21:14:01 dist-test-slave-2kcd krb5kdc[1927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:01.664481 1952 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:01.664922 1952 webserver.cc:460] Webserver started at http://127.27.117.126:37215/ using document root <none> and password file <none>
I20260810 21:14:01.665112 1952 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:01.665153 1952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:01.665251 1952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:01.666206 1952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "5065eba949204febb5517e7253df98d5"
format_stamp: "Formatted at 2026-08-10 21:14:01 on dist-test-slave-2kcd"
server_key: "cb738e82412114c36c42f241204742b6"
server_key_iv: "de2534b5ecb4f1ed3fc5adcb2be12d94"
server_key_version: "encryptionkey@0"
I20260810 21:14:01.666487 1952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "5065eba949204febb5517e7253df98d5"
format_stamp: "Formatted at 2026-08-10 21:14:01 on dist-test-slave-2kcd"
server_key: "cb738e82412114c36c42f241204742b6"
server_key_iv: "de2534b5ecb4f1ed3fc5adcb2be12d94"
server_key_version: "encryptionkey@0"
I20260810 21:14:01.667728 1952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:14:01.668385 1982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:01.668533 1952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:01.668646 1952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "5065eba949204febb5517e7253df98d5"
format_stamp: "Formatted at 2026-08-10 21:14:01 on dist-test-slave-2kcd"
server_key: "cb738e82412114c36c42f241204742b6"
server_key_iv: "de2534b5ecb4f1ed3fc5adcb2be12d94"
server_key_version: "encryptionkey@0"
I20260810 21:14:01.668700 1952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:01.671752 1952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:01.673864 1952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:01.673976 1952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:01.678236 1952 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:43565
I20260810 21:14:01.678303 2057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:43565 every 8 connection(s)
I20260810 21:14:01.678601 1952 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:01.679600 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:01.681753 2058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5: Bootstrap starting.
I20260810 21:14:01.682299 2058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:01.682595 2058 log.cc:824] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:01.683233 2058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5: No bootstrap required, opened a new log
I20260810 21:14:01.684770 2058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065eba949204febb5517e7253df98d5" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43565 } }
I20260810 21:14:01.684995 2058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:14:01.685046 2058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5065eba949204febb5517e7253df98d5, State: Initialized, Role: FOLLOWER
I20260810 21:14:01.685200 2058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065eba949204febb5517e7253df98d5" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43565 } }
I20260810 21:14:01.685297 2058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:14:01.685340 2058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:14:01.685389 2058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:01.686019 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 1952
I20260810 21:14:01.686102 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:01.686265 2058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065eba949204febb5517e7253df98d5" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43565 } }
I20260810 21:14:01.686298 28117 external_mini_cluster.cc:1428] Setting key e159a4a86b0b3ee94668d86b0a6d689c
I20260810 21:14:01.686381 2058 leader_election.cc:302] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5065eba949204febb5517e7253df98d5; no voters:
I20260810 21:14:01.686539 2058 leader_election.cc:288] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:14:01.686580 2062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:01.686808 2058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:01.686874 2062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5065eba949204febb5517e7253df98d5, State: Running, Role: LEADER
I20260810 21:14:01.687050 2062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065eba949204febb5517e7253df98d5" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43565 } }
I20260810 21:14:01.688232 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5065eba949204febb5517e7253df98d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065eba949204febb5517e7253df98d5" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43565 } } }
I20260810 21:14:01.688237 2071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5065eba949204febb5517e7253df98d5. Latest consensus state: current_term: 1 leader_uuid: "5065eba949204febb5517e7253df98d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5065eba949204febb5517e7253df98d5" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 43565 } } }
I20260810 21:14:01.688351 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:01.688370 2071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:01.688683 2089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:14:01.689216 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:01.686731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36893 (local address 127.27.117.126:43565)
0810 21:14:01.686968 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:01.686977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:01.687029 (+ 52us) server_negotiation.cc:407] Connection header received
0810 21:14:01.687145 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:01.687152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:01.687196 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:01.687341 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:01.688673 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:01.688676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:01.688677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:01.688682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:01.688727 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:01.688740 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:01.688784 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:01.688990 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 21:14:01.689034 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":62,"threads_started":1}
I20260810 21:14:01.689507 2089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:01.690164 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:01.691243 2089 catalog_manager.cc:1347] Generated new cluster ID: 243f383af26f48c5882f465aa7e3ad87
I20260810 21:14:01.691294 2089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:01.704720 2089 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:01.704947 2089 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:01.705540 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:01.693188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50773 (local address 127.27.117.126:43565)
0810 21:14:01.693222 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:01.693224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:01.704456 (+ 11232us) server_negotiation.cc:407] Connection header received
0810 21:14:01.704513 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:01.704514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:01.704520 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:01.704576 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:01.705328 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:01.705330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:01.705331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:01.705335 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:01.705348 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:01.705352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:01.705406 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:01.705451 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 21:14:01.705466 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:14:01.710878 2089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5065eba949204febb5517e7253df98d5: Generated new TSK 0
I20260810 21:14:01.711051 2089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:14:01.712488 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 1952
2026-08-10T21:14:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1552 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:14:01 dist-test-slave-2kcd krb5kdc[2114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:14:01 dist-test-slave-2kcd krb5kdc[2114](info): set up 2 sockets
Aug 10 21:14:01 dist-test-slave-2kcd krb5kdc[2114](info): commencing operation
krb5kdc: starting...
W20260810 21:14:03.003491 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:14:03 dist-test-slave-2kcd krb5kdc[2114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:14:03.124922 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:33401
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:45001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:03.208038 2145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:03.208225 2145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:03.208247 2145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:03.209344 2145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:03.209379 2145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:03.209393 2145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:03.209406 2145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:03.209419 2145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:03.210793 2145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:45001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:03.210988 2145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:03.211154 2145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:14:03.216069 2145 server_base.cc:1034] running on GCE node
W20260810 21:14:03.216015 2159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:03.215997 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:03.216045 2157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:03.216566 2145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:03.216841 2145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:03.217999 2145 hybrid_clock.cc:648] HybridClock initialized: now 1786396443217983 us; error 36 us; skew 500 ppm
Aug 10 21:14:03 dist-test-slave-2kcd krb5kdc[2114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396443, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:03.220396 2145 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:03.220861 2145 webserver.cc:460] Webserver started at http://127.27.117.126:41497/ using document root <none> and password file <none>
I20260810 21:14:03.221047 2145 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:03.221087 2145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:03.221170 2145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:03.222054 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "206430ba4637477b9e6dd9bfb0d7e28e"
format_stamp: "Formatted at 2026-08-10 21:14:03 on dist-test-slave-2kcd"
server_key: "8ff6350df90ad1972549f4e159d3a0e6"
server_key_iv: "1210c9360a0f8e08068e8421238150e4"
server_key_version: "encryptionkey@0"
I20260810 21:14:03.222327 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "206430ba4637477b9e6dd9bfb0d7e28e"
format_stamp: "Formatted at 2026-08-10 21:14:03 on dist-test-slave-2kcd"
server_key: "8ff6350df90ad1972549f4e159d3a0e6"
server_key_iv: "1210c9360a0f8e08068e8421238150e4"
server_key_version: "encryptionkey@0"
I20260810 21:14:03.223564 2145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 21:14:03.224256 2171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:03.224443 2145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:14:03.224546 2145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "206430ba4637477b9e6dd9bfb0d7e28e"
format_stamp: "Formatted at 2026-08-10 21:14:03 on dist-test-slave-2kcd"
server_key: "8ff6350df90ad1972549f4e159d3a0e6"
server_key_iv: "1210c9360a0f8e08068e8421238150e4"
server_key_version: "encryptionkey@0"
I20260810 21:14:03.224627 2145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:03.249769 2145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:03.252179 2145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:03.252347 2145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:03.256652 2145 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:33401
I20260810 21:14:03.256708 2257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:33401 every 8 connection(s)
I20260810 21:14:03.257107 2145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:03.257972 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:03.260298 2260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e: Bootstrap starting.
I20260810 21:14:03.260309 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2145
I20260810 21:14:03.260419 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:03.260654 28117 external_mini_cluster.cc:1428] Setting key a5dc1f27d320fbbd0f63decb73f98acc
I20260810 21:14:03.261070 2260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:03.261516 2260 log.cc:824] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:03.262319 2260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e: No bootstrap required, opened a new log
I20260810 21:14:03.264638 2260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33401 } }
I20260810 21:14:03.264824 2260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:14:03.264865 2260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206430ba4637477b9e6dd9bfb0d7e28e, State: Initialized, Role: FOLLOWER
I20260810 21:14:03.265007 2260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33401 } }
I20260810 21:14:03.265095 2260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:14:03.265132 2260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:14:03.265192 2260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:03.265789 2260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33401 } }
I20260810 21:14:03.265908 2260 leader_election.cc:302] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 206430ba4637477b9e6dd9bfb0d7e28e; no voters:
I20260810 21:14:03.266096 2260 leader_election.cc:288] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:14:03.266256 2274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:03.266366 2260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:03.266422 2274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [term 1 LEADER]: Becoming Leader. State: Replica: 206430ba4637477b9e6dd9bfb0d7e28e, State: Running, Role: LEADER
I20260810 21:14:03.266541 2274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33401 } }
I20260810 21:14:03.266985 2275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33401 } } }
I20260810 21:14:03.267154 2275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:03.266964 2274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 206430ba4637477b9e6dd9bfb0d7e28e. Latest consensus state: current_term: 1 leader_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206430ba4637477b9e6dd9bfb0d7e28e" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 33401 } } }
I20260810 21:14:03.267429 2274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:03.267902 2289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 21:14:03 dist-test-slave-2kcd krb5kdc[2114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:14:03.268400 2289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:03.269973 2289 catalog_manager.cc:1347] Generated new cluster ID: 6165019ac55f4b4595064e7e3abc7073
I20260810 21:14:03.270030 2289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:03.272635 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:03.261254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36781 (local address 127.27.117.126:33401)
0810 21:14:03.261420 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:03.261430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:03.261474 (+ 44us) server_negotiation.cc:407] Connection header received
0810 21:14:03.262100 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:03.262107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:03.262154 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:03.262266 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:03.263077 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:03.263890 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:03.265195 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:03.265657 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:03.265739 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:03.268574 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:03.268578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:03.268578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:03.268594 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:03.270858 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:03.271230 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:03.271232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:03.271233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:03.271321 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:03.271488 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:03.271488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:03.271489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:03.271738 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:03.272233 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:03.272315 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 21:14:03.272433 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260810 21:14:03.273381 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:03.275790 2289 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:03.276463 2289 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:03.281101 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:03.274806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57897 (local address 127.27.117.126:33401)
0810 21:14:03.274838 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:03.274839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:03.274975 (+ 136us) server_negotiation.cc:407] Connection header received
0810 21:14:03.275030 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:03.275031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:03.275042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:03.275103 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:03.275810 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:03.276312 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:03.277126 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:03.277363 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:03.277436 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:03.278575 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:03.278576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:03.278576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:03.278583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:03.279933 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:03.280270 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:03.280271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:03.280272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:03.280325 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:03.280540 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:03.280540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:03.280541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:03.280647 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:03.280758 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:03.280897 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 21:14:03.280986 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 21:14:03.283418 2289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 206430ba4637477b9e6dd9bfb0d7e28e: Generated new TSK 0
I20260810 21:14:03.283571 2289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:14:03.285434 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2145
2026-08-10T21:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1568 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:14:03 dist-test-slave-2kcd krb5kdc[2324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:14:03 dist-test-slave-2kcd krb5kdc[2324](info): set up 2 sockets
Aug 10 21:14:03 dist-test-slave-2kcd krb5kdc[2324](info): commencing operation
krb5kdc: starting...
W20260810 21:14:04.585417 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:14:04 dist-test-slave-2kcd krb5kdc[2324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:14:04.705260 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:46833
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:36099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:04.787994 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:04.788225 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:04.788266 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:04.789498 2342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:04.789566 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:04.789593 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:04.789623 2342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:04.789647 2342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:04.791092 2342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:36099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:46833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:04.791365 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:04.791594 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 21:14:04.796525 2342 server_base.cc:1034] running on GCE node
W20260810 21:14:04.796480 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:04.796483 2350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:04.796460 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:04.796952 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:04.797230 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:04.798373 2342 hybrid_clock.cc:648] HybridClock initialized: now 1786396444798361 us; error 29 us; skew 500 ppm
Aug 10 21:14:04 dist-test-slave-2kcd krb5kdc[2324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396444, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:04.800904 2342 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:04.801333 2342 webserver.cc:460] Webserver started at http://127.27.117.126:41801/ using document root <none> and password file <none>
I20260810 21:14:04.801568 2342 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:04.801614 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:04.801759 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:04.802793 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "4a7a089f1822475badcd4098aec26e2b"
format_stamp: "Formatted at 2026-08-10 21:14:04 on dist-test-slave-2kcd"
server_key: "2155cf2b8307054db0f37214c8d7ae45"
server_key_iv: "b83ddb522bf62b33a3106bad76bab280"
server_key_version: "encryptionkey@0"
I20260810 21:14:04.803177 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "4a7a089f1822475badcd4098aec26e2b"
format_stamp: "Formatted at 2026-08-10 21:14:04 on dist-test-slave-2kcd"
server_key: "2155cf2b8307054db0f37214c8d7ae45"
server_key_iv: "b83ddb522bf62b33a3106bad76bab280"
server_key_version: "encryptionkey@0"
I20260810 21:14:04.804519 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:14:04.805411 2365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:04.805606 2342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:14:04.805717 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "4a7a089f1822475badcd4098aec26e2b"
format_stamp: "Formatted at 2026-08-10 21:14:04 on dist-test-slave-2kcd"
server_key: "2155cf2b8307054db0f37214c8d7ae45"
server_key_iv: "b83ddb522bf62b33a3106bad76bab280"
server_key_version: "encryptionkey@0"
I20260810 21:14:04.805804 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:04.817040 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:04.818817 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:04.818981 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:04.823114 2342 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:46833
I20260810 21:14:04.823146 2448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:46833 every 8 connection(s)
I20260810 21:14:04.823494 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:04.824332 2449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:04.826676 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b: Bootstrap starting.
I20260810 21:14:04.827266 2449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:04.827580 2449 log.cc:824] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:04.828269 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b: No bootstrap required, opened a new log
I20260810 21:14:04.830020 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2342
I20260810 21:14:04.830164 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:04.830080 2449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7a089f1822475badcd4098aec26e2b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 46833 } }
I20260810 21:14:04.830287 2449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:14:04.830350 2449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7a089f1822475badcd4098aec26e2b, State: Initialized, Role: FOLLOWER
I20260810 21:14:04.830402 28117 external_mini_cluster.cc:1428] Setting key 0b7fe501a92d2f679ad9583ee2fd846f
I20260810 21:14:04.830509 2449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7a089f1822475badcd4098aec26e2b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 46833 } }
I20260810 21:14:04.830600 2449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:14:04.830636 2449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:14:04.830688 2449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:04.831833 2449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7a089f1822475badcd4098aec26e2b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 46833 } }
I20260810 21:14:04.831944 2449 leader_election.cc:302] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a7a089f1822475badcd4098aec26e2b; no voters:
I20260810 21:14:04.832098 2449 leader_election.cc:288] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:14:04.832160 2458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:04.832341 2449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:04.832347 2458 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [term 1 LEADER]: Becoming Leader. State: Replica: 4a7a089f1822475badcd4098aec26e2b, State: Running, Role: LEADER
I20260810 21:14:04.832543 2458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7a089f1822475badcd4098aec26e2b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 46833 } }
I20260810 21:14:04.832914 2460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a7a089f1822475badcd4098aec26e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7a089f1822475badcd4098aec26e2b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 46833 } } }
I20260810 21:14:04.832950 2461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a7a089f1822475badcd4098aec26e2b. Latest consensus state: current_term: 1 leader_uuid: "4a7a089f1822475badcd4098aec26e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a7a089f1822475badcd4098aec26e2b" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 46833 } } }
I20260810 21:14:04.833050 2460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:04.833114 2461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:04.833470 2476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:14:04.834064 2476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:04.836148 2476 catalog_manager.cc:1347] Generated new cluster ID: 50b76cdbbd834a98b8da63a923f0d9d1
I20260810 21:14:04.836236 2476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:04.836612 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:04.830879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57889 (local address 127.27.117.126:46833)
0810 21:14:04.831094 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:04.831104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:04.831137 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:14:04.831271 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:04.831281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:04.831345 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:04.831513 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:04.831695 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:04.833152 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:04.834360 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:04.835000 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:04.835106 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:04.835877 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:04.835881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:04.835881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:04.836059 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:04.836090 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:04.836101 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:04.836181 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:04.836370 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 21:14:04.836441 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260810 21:14:04.837531 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:04.844026 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:04.838932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33387 (local address 127.27.117.126:46833)
0810 21:14:04.838975 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:04.838977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:04.841059 (+ 2082us) server_negotiation.cc:407] Connection header received
0810 21:14:04.841142 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:04.841143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:04.841169 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:04.841233 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:04.841393 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:04.841907 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:04.842679 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:04.842897 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:04.842991 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:04.843710 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:04.843712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:04.843712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:04.843717 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:04.843733 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:04.843737 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:04.843829 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:04.843898 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 21:14:04.843924 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 21:14:04.853734 2476 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:04.854434 2476 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:04.869896 2476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a7a089f1822475badcd4098aec26e2b: Generated new TSK 0
I20260810 21:14:04.870078 2476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:14:04.874743 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2342
2026-08-10T21:14:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1584 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:14:04 dist-test-slave-2kcd krb5kdc[2500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:14:04 dist-test-slave-2kcd krb5kdc[2500](info): set up 2 sockets
Aug 10 21:14:04 dist-test-slave-2kcd krb5kdc[2500](info): commencing operation
krb5kdc: starting...
W20260810 21:14:06.160177 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:14:06 dist-test-slave-2kcd krb5kdc[2500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:14:06.274681 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41537
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:46343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:06.354787 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:06.354977 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:06.355000 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:06.356592 2525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:06.356637 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:06.356650 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:06.356663 2525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:06.356675 2525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:06.358114 2525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:46343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:06.358373 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:06.358616 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:06.363401 2533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:06.363411 2535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:06.363436 2539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:06.363428 2525 server_base.cc:1034] running on GCE node
I20260810 21:14:06.363814 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:06.364120 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:06.365283 2525 hybrid_clock.cc:648] HybridClock initialized: now 1786396446365267 us; error 30 us; skew 500 ppm
Aug 10 21:14:06 dist-test-slave-2kcd krb5kdc[2500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396446, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:06.373078 2525 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:06.373553 2525 webserver.cc:460] Webserver started at http://127.27.117.126:42851/ using document root <none> and password file <none>
I20260810 21:14:06.373798 2525 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:06.373869 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:06.374033 2525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:06.375442 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "619a98db0ff4493fa9175b5afec5b950"
format_stamp: "Formatted at 2026-08-10 21:14:06 on dist-test-slave-2kcd"
server_key: "dd3c9b2f71bfdb891bfba3e38fa6b0c2"
server_key_iv: "a803fad474d47a7d7d0e2ef74cee8f21"
server_key_version: "encryptionkey@0"
I20260810 21:14:06.375905 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "619a98db0ff4493fa9175b5afec5b950"
format_stamp: "Formatted at 2026-08-10 21:14:06 on dist-test-slave-2kcd"
server_key: "dd3c9b2f71bfdb891bfba3e38fa6b0c2"
server_key_iv: "a803fad474d47a7d7d0e2ef74cee8f21"
server_key_version: "encryptionkey@0"
I20260810 21:14:06.377285 2525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:14:06.377975 2555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:06.378158 2525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:14:06.378252 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "619a98db0ff4493fa9175b5afec5b950"
format_stamp: "Formatted at 2026-08-10 21:14:06 on dist-test-slave-2kcd"
server_key: "dd3c9b2f71bfdb891bfba3e38fa6b0c2"
server_key_iv: "a803fad474d47a7d7d0e2ef74cee8f21"
server_key_version: "encryptionkey@0"
I20260810 21:14:06.378332 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:06.381619 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:06.383363 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:06.383520 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:06.387662 2525 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:41537
I20260810 21:14:06.387696 2650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:41537 every 8 connection(s)
I20260810 21:14:06.388005 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:06.388669 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2525
I20260810 21:14:06.388814 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:06.389005 28117 external_mini_cluster.cc:1428] Setting key f716b1055b95f1a331d189c9a58c9ae8
I20260810 21:14:06.389065 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:06.391390 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:06.389499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44743 (local address 127.27.117.126:41537)
0810 21:14:06.389692 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:06.389700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:06.389735 (+ 35us) server_negotiation.cc:407] Connection header received
0810 21:14:06.390160 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:06.390165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:06.390189 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:06.390297 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:06.390935 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:06.390939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:06.390939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:06.390942 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:06.390977 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:06.390986 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:06.391048 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:06.391133 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 21:14:06.391161 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":109,"threads_started":1}
I20260810 21:14:06.392458 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:06.392704 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950: Bootstrap starting.
I20260810 21:14:06.393316 2653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:06.393662 2653 log.cc:824] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:06.394484 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950: No bootstrap required, opened a new log
I20260810 21:14:06.395017 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:06.393931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54631 (local address 127.27.117.126:41537)
0810 21:14:06.393973 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:06.393975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:06.394136 (+ 161us) server_negotiation.cc:407] Connection header received
0810 21:14:06.394184 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:06.394185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:06.394189 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:06.394258 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:06.394790 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:06.394791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:06.394791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 21:14:06.394795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:06.394807 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 21:14:06.394811 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:06.394859 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:06.394920 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 21:14:06.394930 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:14:06.396618 2653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619a98db0ff4493fa9175b5afec5b950" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41537 } }
I20260810 21:14:06.396807 2653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 21:14:06.396852 2653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619a98db0ff4493fa9175b5afec5b950, State: Initialized, Role: FOLLOWER
I20260810 21:14:06.396966 2653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619a98db0ff4493fa9175b5afec5b950" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41537 } }
I20260810 21:14:06.397048 2653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 21:14:06.397109 2653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 21:14:06.397157 2653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:06.397667 2653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619a98db0ff4493fa9175b5afec5b950" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41537 } }
I20260810 21:14:06.397781 2653 leader_election.cc:302] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 619a98db0ff4493fa9175b5afec5b950; no voters:
I20260810 21:14:06.397967 2653 leader_election.cc:288] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 21:14:06.398010 2677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:06.398182 2677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [term 1 LEADER]: Becoming Leader. State: Replica: 619a98db0ff4493fa9175b5afec5b950, State: Running, Role: LEADER
I20260810 21:14:06.398226 2653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:06.398366 2677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619a98db0ff4493fa9175b5afec5b950" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41537 } }
I20260810 21:14:06.398763 2681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 619a98db0ff4493fa9175b5afec5b950. Latest consensus state: current_term: 1 leader_uuid: "619a98db0ff4493fa9175b5afec5b950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619a98db0ff4493fa9175b5afec5b950" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41537 } } }
I20260810 21:14:06.398888 2681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:06.399082 2692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:14:06.398756 2679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "619a98db0ff4493fa9175b5afec5b950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "619a98db0ff4493fa9175b5afec5b950" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41537 } } }
I20260810 21:14:06.399443 2679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:06.399616 2692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:06.401108 2692 catalog_manager.cc:1347] Generated new cluster ID: d00a28ba77634b29ace9924bca09e250
I20260810 21:14:06.401181 2692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:06.420146 2692 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:06.420325 2692 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:06.426227 2692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 619a98db0ff4493fa9175b5afec5b950: Generated new TSK 0
I20260810 21:14:06.426419 2692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 21:14:06.433602 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2525
2026-08-10T21:14:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1555 ms)
[----------] 18 tests from AuthTokenIssuingTest (28120 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:14:06 dist-test-slave-2kcd krb5kdc[2715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:14:06 dist-test-slave-2kcd krb5kdc[2715](info): set up 2 sockets
Aug 10 21:14:06 dist-test-slave-2kcd krb5kdc[2715](info): commencing operation
krb5kdc: starting...
W20260810 21:14:07.703975 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:14:07 dist-test-slave-2kcd krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:14:07.825687 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41009
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41009,127.27.117.125:39851,127.27.117.124:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:07.908282 2732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 21:14:07.908497 2732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:07.908515 2732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:07.908535 2732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:07.909724 2732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:07.909766 2732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:07.909781 2732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:07.909794 2732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:07.909806 2732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:07.911219 2732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41009,127.27.117.125:39851,127.27.117.124:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:07.911406 2732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:07.911585 2732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:07.916293 2748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:07.916306 2751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:07.916306 2745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:07.916584 2732 server_base.cc:1034] running on GCE node
I20260810 21:14:07.916867 2732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:07.917179 2732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:07.918340 2732 hybrid_clock.cc:648] HybridClock initialized: now 1786396447918335 us; error 44 us; skew 500 ppm
Aug 10 21:14:07 dist-test-slave-2kcd krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:07.920959 2732 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:07.921373 2732 webserver.cc:460] Webserver started at http://127.27.117.126:40497/ using document root <none> and password file <none>
I20260810 21:14:07.921583 2732 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:07.921628 2732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:07.921722 2732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:07.922730 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "53ed75047f1f461eadef81d4d3c45222"
format_stamp: "Formatted at 2026-08-10 21:14:07 on dist-test-slave-2kcd"
server_key: "c623212c19b5cf30c83d467854f3e920"
server_key_iv: "836696928869b5e41f05221c61f86149"
server_key_version: "encryptionkey@0"
I20260810 21:14:07.923044 2732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "53ed75047f1f461eadef81d4d3c45222"
format_stamp: "Formatted at 2026-08-10 21:14:07 on dist-test-slave-2kcd"
server_key: "c623212c19b5cf30c83d467854f3e920"
server_key_iv: "836696928869b5e41f05221c61f86149"
server_key_version: "encryptionkey@0"
I20260810 21:14:07.924378 2732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 21:14:07.925130 2764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:07.925307 2732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 21:14:07.925395 2732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "53ed75047f1f461eadef81d4d3c45222"
format_stamp: "Formatted at 2026-08-10 21:14:07 on dist-test-slave-2kcd"
server_key: "c623212c19b5cf30c83d467854f3e920"
server_key_iv: "836696928869b5e41f05221c61f86149"
server_key_version: "encryptionkey@0"
I20260810 21:14:07.925449 2732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:07.941877 2732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:07.943598 2732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:07.943706 2732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:07.950896 2732 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:41009
I20260810 21:14:07.951092 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:41009 every 8 connection(s)
I20260810 21:14:07.952111 2732 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:07.954665 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:07.960206 2874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:07.961077 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2732
I20260810 21:14:07.961414 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:07.961855 28117 external_mini_cluster.cc:1428] Setting key ec090b06339fe51ae2176c527ed9c30a
Aug 10 21:14:07 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.125@KRBTEST.COM; defaulting to no policy
I20260810 21:14:07.993777 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:07.962294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.27.117.126:41009)
0810 21:14:07.963076 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:07.963146 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:07.963483 (+ 337us) server_negotiation.cc:407] Connection header received
0810 21:14:07.965255 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:07.965278 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:07.965320 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:07.965471 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:07.967078 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:07.969763 (+ 2685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:07.973023 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:07.973780 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:07.974094 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:07.981313 (+ 7219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:07.981333 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:07.981333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:07.981357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:07.989255 (+ 7898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:07.990499 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:07.990542 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:07.990553 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:07.990880 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:07.991456 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:07.991457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:07.991458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:07.992008 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:07.992148 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:07.993041 (+ 893us) server_negotiation.cc:299] Negotiation successful
0810 21:14:07.993298 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":563,"thread_start_us":330,"threads_started":1}
I20260810 21:14:07.993779 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:07.962286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41009 (local address 127.0.0.1:41190)
0810 21:14:07.963101 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:07.963253 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:07.964509 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:07.965494 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:07.965497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:07.966250 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:07.966809 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:07.966813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:07.970093 (+ 3280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:07.970094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:07.972740 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:07.972744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:07.974150 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:07.974152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:07.974502 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:07.976066 (+ 1564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:07.976099 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:07.980992 (+ 4893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:07.989623 (+ 8631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:07.989645 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:07.989656 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:07.990174 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:07.991068 (+ 894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:07.991072 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:07.991072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:07.991290 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:07.992127 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:07.992139 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:07.992366 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 21:14:07.993008 (+ 642us) client_negotiation.cc:240] Negotiation successful
0810 21:14:07.993376 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":127,"thread_start_us":334,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.125@KRBTEST.COM" created.
I20260810 21:14:07.996637 2874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:07.997918 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:07.997409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48804)
0810 21:14:07.997549 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:07.997745 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 21:14:07.998415 2766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
I20260810 21:14:07.999572 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:07.999290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48806)
0810 21:14:07.999384 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:07.999423 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 21:14:07.999994 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:39851: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab.
Entry for principal kudu/127.27.117.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab.
I20260810 21:14:08.029966 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:39851
--webserver_interface=127.27.117.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab
--principal=kudu/127.27.117.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41009,127.27.117.125:39851,127.27.117.124:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:14:08.049511 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } attempt: 1
I20260810 21:14:08.051523 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.051016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48820)
0810 21:14:08.051190 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.051337 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 21:14:08.052755 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.052468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48822)
0810 21:14:08.052597 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.052710 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 21:14:08.053200 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:39851: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
I20260810 21:14:08.121464 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } attempt: 2
I20260810 21:14:08.122218 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.121964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48834)
0810 21:14:08.122135 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.122190 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 21:14:08.122886 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.122601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48840)
0810 21:14:08.122684 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.122708 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 21:14:08.123235 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:39851: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
I20260810 21:14:08.214665 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } attempt: 3
I20260810 21:14:08.216104 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.215697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48850)
0810 21:14:08.215906 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.215975 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260810 21:14:08.216920 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.216709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48854)
0810 21:14:08.216834 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.216890 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 21:14:08.217067 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:39851: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
W20260810 21:14:08.316270 2884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 21:14:08.316672 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:08.316711 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:08.316747 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:08.321352 2884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:08.321528 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:08.321763 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:08.321799 2884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:08.321943 2884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:08.327375 2884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41009,127.27.117.125:39851,127.27.117.124:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab
--principal=kudu/127.27.117.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:39851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.117.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:08.328395 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:08.329391 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:08.342774 2898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:08.343217 2901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:08.342811 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:08.344053 2884 server_base.cc:1034] running on GCE node
I20260810 21:14:08.344717 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:08.345767 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:08.347146 2884 hybrid_clock.cc:648] HybridClock initialized: now 1786396448347092 us; error 66 us; skew 500 ppm
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:08.354573 2884 init.cc:377] Logged in from keytab as kudu/127.27.117.125@KRBTEST.COM (short username kudu)
I20260810 21:14:08.355525 2884 webserver.cc:460] Webserver started at http://127.27.117.125:44823/ using document root <none> and password file <none>
I20260810 21:14:08.356158 2884 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:08.356285 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:08.356593 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:08.358637 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data/instance:
uuid: "4f7139fd3ace478e805a8e26e451b79d"
format_stamp: "Formatted at 2026-08-10 21:14:08 on dist-test-slave-2kcd"
server_key: "4db9df61c535a2c5961d97e0965ef7d5"
server_key_iv: "c0380401627b317e6c83819a29503132"
server_key_version: "encryptionkey@0"
I20260810 21:14:08.359336 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal/instance:
uuid: "4f7139fd3ace478e805a8e26e451b79d"
format_stamp: "Formatted at 2026-08-10 21:14:08 on dist-test-slave-2kcd"
server_key: "4db9df61c535a2c5961d97e0965ef7d5"
server_key_iv: "c0380401627b317e6c83819a29503132"
server_key_version: "encryptionkey@0"
I20260810 21:14:08.360241 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } attempt: 4
I20260810 21:14:08.360896 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.360690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48860)
0810 21:14:08.360811 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.360858 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 21:14:08.361490 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.361330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48868)
0810 21:14:08.361399 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.361445 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 21:14:08.361616 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:39851: Network error: Client connection negotiation failed: client connection to 127.27.117.125:39851: connect: Connection refused (error 111)
I20260810 21:14:08.362450 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 21:14:08.363875 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:08.364224 2884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 21:14:08.364427 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal
uuid: "4f7139fd3ace478e805a8e26e451b79d"
format_stamp: "Formatted at 2026-08-10 21:14:08 on dist-test-slave-2kcd"
server_key: "4db9df61c535a2c5961d97e0965ef7d5"
server_key_iv: "c0380401627b317e6c83819a29503132"
server_key_version: "encryptionkey@0"
I20260810 21:14:08.364578 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:08.437974 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:08.443629 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:08.443945 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:08.452427 2884 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.125:39851
I20260810 21:14:08.452495 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.125:39851 every 8 connection(s)
I20260810 21:14:08.453186 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
I20260810 21:14:08.454710 3004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:08.458801 3004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:08.461071 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 2884
I20260810 21:14:08.461340 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-1/wal/instance
I20260810 21:14:08.461752 28117 external_mini_cluster.cc:1428] Setting key 6793f54bef1f88efbc37bdcabc74ddff
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.124@KRBTEST.COM; defaulting to no policy
I20260810 21:14:08.484282 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.460637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.27.117.126:41009)
0810 21:14:08.460877 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.460885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.461444 (+ 559us) server_negotiation.cc:407] Connection header received
0810 21:14:08.463171 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.463172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.463196 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.463268 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.465327 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.466794 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.469290 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.469599 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.469689 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.476375 (+ 6686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.476377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.476379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.476391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.481695 (+ 5304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.482519 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.482519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.482520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.482614 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.483122 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.483123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.483123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.483302 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.483429 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.483907 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.484140 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 21:14:08.484602 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.460619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41009 (local address 127.0.0.1:41200)
0810 21:14:08.461087 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.461273 (+ 186us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.462952 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.463484 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.463490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.464301 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.465094 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.465113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.466982 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.466983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.469157 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.469160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.469736 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.469738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.470019 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.471411 (+ 1392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.471486 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.475943 (+ 4457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.481873 (+ 5930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.481897 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.481899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.482307 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.482741 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.482742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.482743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.482932 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.483416 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.483423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.483650 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.483851 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.484191 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":361,"threads_started":1}
I20260810 21:14:08.485543 3004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.124@KRBTEST.COM" created.
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:14:08.502584 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.486059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48874)
0810 21:14:08.486156 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.486179 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.486260 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.486699 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.486700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.486911 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.486980 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.486982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.489307 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.489308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.490654 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.490666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.491495 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.491496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.491671 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.492320 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.492326 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.496158 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.499860 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.499861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.499861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.500265 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.500681 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.500681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.500682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.500846 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.501341 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.501342 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.501422 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.501730 (+ 308us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.502129 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 21:14:08.502767 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.486091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.27.117.125:39851)
0810 21:14:08.486410 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.486416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.486472 (+ 56us) server_negotiation.cc:407] Connection header received
0810 21:14:08.486519 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.486532 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.486560 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.486713 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.487086 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.489111 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.490871 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.491345 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.491464 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.496456 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.496490 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.496490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.496512 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.499636 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.500446 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.500450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.500451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.500562 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.500928 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.500928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.500929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.501178 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.501378 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.501887 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.502249 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":224,"server-negotiator.queue_time_us":250,"thread_start_us":164,"threads_started":1}
I20260810 21:14:08.504722 3004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:08.505513 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.505344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47530)
0810 21:14:08.505384 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.505470 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260810 21:14:08.505919 2915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
I20260810 21:14:08.506850 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.506642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47536)
0810 21:14:08.506737 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.506761 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 21:14:08.507154 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab.
Entry for principal kudu/127.27.117.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab.
I20260810 21:14:08.514827 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:41577
--webserver_interface=127.27.117.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab
--principal=kudu/127.27.117.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:41009,127.27.117.125:39851,127.27.117.124:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:14:08.556659 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 1
I20260810 21:14:08.558048 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.557596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47548)
0810 21:14:08.557823 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.557902 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260810 21:14:08.559307 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.558882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47560)
0810 21:14:08.559051 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.559238 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 21:14:08.559746 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
I20260810 21:14:08.628093 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 2
I20260810 21:14:08.629242 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.628998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47574)
0810 21:14:08.629105 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.629185 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 21:14:08.629874 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.629759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47590)
0810 21:14:08.629822 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.629848 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 21:14:08.630102 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
I20260810 21:14:08.660920 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } attempt: 5
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:14:08.682389 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.662025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48890)
0810 21:14:08.662219 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.662240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.662431 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.663147 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.663147 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.663368 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.663468 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.663470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.665226 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.665227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.665907 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.665909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.666631 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.666632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.666705 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.667576 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.667589 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.672267 (+ 4678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.679757 (+ 7490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.679758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.679758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.680199 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.680930 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.680930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.680930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.681012 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.681973 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.681974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.682100 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.682231 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.682318 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 21:14:08.682461 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.662279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.27.117.125:39851)
0810 21:14:08.662487 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.662507 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.662659 (+ 152us) server_negotiation.cc:407] Connection header received
0810 21:14:08.662757 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.662767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.662866 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.663137 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.663587 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.665118 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.666018 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.666525 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.666626 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.672556 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.672558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.672558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.672573 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.679435 (+ 6862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.680413 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.680415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.680424 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.680667 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.681107 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.681108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.681108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.681706 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.681996 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.682272 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.682366 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260810 21:14:08.683650 2874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:08.684442 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.684183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47592)
0810 21:14:08.684264 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.684355 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 21:14:08.685410 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.685233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47604)
0810 21:14:08.685328 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.685361 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 21:14:08.685608 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
I20260810 21:14:08.721427 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 3
I20260810 21:14:08.722410 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.722169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47620)
0810 21:14:08.722255 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.722320 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 21:14:08.723078 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.722956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47630)
0810 21:14:08.723011 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.723061 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 21:14:08.723259 3004 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
I20260810 21:14:08.736943 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 1
I20260810 21:14:08.737891 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.737561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47646)
0810 21:14:08.737724 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.737859 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 21:14:08.739010 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.738691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47652)
0810 21:14:08.738865 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.738958 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260810 21:14:08.739284 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
W20260810 21:14:08.750335 3014 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 21:14:08.750862 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:08.750949 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:08.751010 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:08.753563 3014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:08.753625 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:08.753654 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:08.753674 3014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:08.753695 3014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:08.755970 3014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:41009,127.27.117.125:39851,127.27.117.124:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab
--principal=kudu/127.27.117.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.117.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:08.756362 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:08.756696 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:08.764377 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:08.764393 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:08.764413 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:08.764715 3014 server_base.cc:1034] running on GCE node
I20260810 21:14:08.765177 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:08.765626 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:08.766805 3014 hybrid_clock.cc:648] HybridClock initialized: now 1786396448766792 us; error 64 us; skew 500 ppm
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:08.770828 3014 init.cc:377] Logged in from keytab as kudu/127.27.117.124@KRBTEST.COM (short username kudu)
I20260810 21:14:08.771675 3014 webserver.cc:460] Webserver started at http://127.27.117.124:37233/ using document root <none> and password file <none>
I20260810 21:14:08.772050 3014 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:08.772161 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:08.772394 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:08.774111 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data/instance:
uuid: "a7647e7bb2ac4f69b98697d808ae49ae"
format_stamp: "Formatted at 2026-08-10 21:14:08 on dist-test-slave-2kcd"
server_key: "987feffe95918c525e765b2431ddb393"
server_key_iv: "7cbf280f8695f43a1595bab48fe4b711"
server_key_version: "encryptionkey@0"
I20260810 21:14:08.774968 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal/instance:
uuid: "a7647e7bb2ac4f69b98697d808ae49ae"
format_stamp: "Formatted at 2026-08-10 21:14:08 on dist-test-slave-2kcd"
server_key: "987feffe95918c525e765b2431ddb393"
server_key_iv: "7cbf280f8695f43a1595bab48fe4b711"
server_key_version: "encryptionkey@0"
I20260810 21:14:08.779247 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 21:14:08.781520 3050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:08.782073 3014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:14:08.782378 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal
uuid: "a7647e7bb2ac4f69b98697d808ae49ae"
format_stamp: "Formatted at 2026-08-10 21:14:08 on dist-test-slave-2kcd"
server_key: "987feffe95918c525e765b2431ddb393"
server_key_iv: "7cbf280f8695f43a1595bab48fe4b711"
server_key_version: "encryptionkey@0"
I20260810 21:14:08.782609 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:08.807560 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 2
I20260810 21:14:08.808861 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.808416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47666)
0810 21:14:08.808540 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.808729 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 21:14:08.809695 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.809532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47672)
0810 21:14:08.809627 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.809667 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 21:14:08.810046 2874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:41577: Network error: Client connection negotiation failed: client connection to 127.27.117.124:41577: connect: Connection refused (error 111)
I20260810 21:14:08.835883 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:08.842010 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:08.842206 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:08.849933 3014 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.124:41577
I20260810 21:14:08.850062 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.124:41577 every 8 connection(s)
I20260810 21:14:08.851236 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
I20260810 21:14:08.853207 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3014
I20260810 21:14:08.853574 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786396166878326-28117-0/minicluster-data/master-2/wal/instance
I20260810 21:14:08.853925 3144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:08.854288 28117 external_mini_cluster.cc:1428] Setting key b255c5d4bfbba678745c710e1bf799b9
I20260810 21:14:08.861089 3144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:08.866492 3004 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 4
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:14:08.885872 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.862924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.27.117.126:41009)
0810 21:14:08.863066 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.863068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.863512 (+ 444us) server_negotiation.cc:407] Connection header received
0810 21:14:08.863605 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.863607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.863630 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.863739 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.864840 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.865534 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.867781 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.868240 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.868329 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.876861 (+ 8532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.876873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.876873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.876921 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.881670 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.883017 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.883019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.883029 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.883155 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.883880 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.883881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.883881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.884150 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.884377 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.885265 (+ 888us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.885518 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 21:14:08.886921 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.862862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:41009 (local address 127.0.0.1:41208)
0810 21:14:08.863236 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.863311 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.863476 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.863850 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.863853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.864511 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.864674 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.864679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.865664 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.865666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.867614 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.867617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.868344 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.868346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.868599 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.870103 (+ 1504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.870138 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.876437 (+ 6299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.881910 (+ 5473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.881925 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.881927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.882728 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.883382 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.883388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.883388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.883711 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.884377 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.884386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.884723 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.885188 (+ 465us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.885788 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":278,"threads_started":1}
I20260810 21:14:08.888187 3144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:08.888667 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.856658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.27.117.124:41577)
0810 21:14:08.857243 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.857253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.857343 (+ 90us) server_negotiation.cc:407] Connection header received
0810 21:14:08.858912 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.858961 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.859149 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.859750 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.862311 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.864879 (+ 2568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.866260 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.868060 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.868166 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.875849 (+ 7683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.875885 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.875885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.875914 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.883922 (+ 8008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.885570 (+ 1648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.885575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.885576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.885849 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.886647 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.886657 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.886657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.887176 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.888183 (+ 1007us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.888246 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.888441 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":384,"threads_started":1}
I20260810 21:14:08.889981 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.867112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47688)
0810 21:14:08.867209 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.867265 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.867466 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.868070 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.868071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.868254 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.868304 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.868306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.870536 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.870537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.872583 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.872596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.873772 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.873773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.873888 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.874805 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.874833 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.879091 (+ 4258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.887806 (+ 8715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.887808 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.887808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.888225 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.888674 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.888675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.888676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.888852 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.889538 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.889539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.889609 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.889723 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.889880 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 21:14:08.890645 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.867203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.27.117.124:41577)
0810 21:14:08.867693 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.867694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.867714 (+ 20us) server_negotiation.cc:407] Connection header received
0810 21:14:08.867755 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.867756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.867810 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.868060 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.868434 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.870182 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.873016 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.873606 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.873772 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.879435 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.879437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.879437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.879454 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.887656 (+ 8202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.888420 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.888421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.888421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.888489 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.889000 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.889001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.889001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.889358 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.890164 (+ 806us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.890298 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.890451 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":357,"threads_started":1}
I20260810 21:14:08.895303 3004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: Bootstrap starting.
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:14:08.898407 3004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:08.899531 3004 log.cc:824] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:08.901340 3004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: No bootstrap required, opened a new log
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:14:08.906069 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.888832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.27.117.125:39851)
0810 21:14:08.888922 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.888926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.889052 (+ 126us) server_negotiation.cc:407] Connection header received
0810 21:14:08.889393 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.889394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.889416 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.889546 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.890792 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.891715 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.893278 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.893725 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.893906 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.898229 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.898230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.898230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.898248 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.902050 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.903492 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.903493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.903493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.903621 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.904150 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.904151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.904151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.904665 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.904990 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.905584 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.905753 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 21:14:08.906070 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.888706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:39851 (local address 127.0.0.1:48898)
0810 21:14:08.888966 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.888988 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.889087 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.889553 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.889555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.889841 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.889970 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.889973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.891995 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.891996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.893022 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.893024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.893906 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.893916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.894121 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.895062 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.895088 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.898024 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.902312 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.902314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.902315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.903156 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.903826 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.903832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.903832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.903986 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.904975 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.904977 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.905158 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.905372 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.905731 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260810 21:14:08.907303 3144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:08.909296 3004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.909885 3004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7139fd3ace478e805a8e26e451b79d, State: Initialized, Role: FOLLOWER
I20260810 21:14:08.910249 3004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.911468 3004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:08.911224 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:08.911736 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:08.915481 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.890372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.27.117.126:41009)
0810 21:14:08.890442 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.890444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.890452 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:14:08.890524 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.890525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.890535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.890639 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.891483 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.892445 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.894552 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.895292 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.895471 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.903076 (+ 7605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.903087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.903087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.903133 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.910672 (+ 7539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.912582 (+ 1910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.912593 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.912594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.912832 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.913837 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.913839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.913839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.914315 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.914855 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.915006 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.915170 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:14:08.916218 2874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } attempt: 3
W20260810 21:14:08.917135 3183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:08.917217 3183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396448, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:14:08.931851 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.908236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47692)
0810 21:14:08.908360 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.908372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.908533 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.909027 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.909028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.909452 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.909485 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.909487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.911645 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.911652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.913841 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.913849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.915538 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.915539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.915679 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.916459 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.916464 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.920065 (+ 3601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.928261 (+ 8196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.928263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.928263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.929203 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.930137 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.930149 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.930149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.930482 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.931401 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.931402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.931441 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.931494 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.931590 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 21:14:08.933749 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.908385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.27.117.124:41577)
0810 21:14:08.908549 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.908571 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.908600 (+ 29us) server_negotiation.cc:407] Connection header received
0810 21:14:08.908730 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.908731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.908752 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.909027 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.909810 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.911357 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.914221 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.915136 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.915437 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.920286 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.920287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.920288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.920354 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.927913 (+ 7559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.929651 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.929660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.929661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.929824 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.930703 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.930712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.930713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.931299 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.931377 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.933071 (+ 1694us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.933416 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:14:08.938103 3144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: Bootstrap starting.
Aug 10 21:14:08 dist-test-slave-2kcd krb5kdc[2715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396447, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:14:08.940022 3144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:08.941540 3144 log.cc:824] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:08.924491 2884 master_runner.cc:186] Error getting master registration for 127.27.117.126:41009: OK, instance_id { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" instance_seqno: 1786396447947752 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 21:14:08.942126 2884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 21:14:08.945829 2926 tablet.cc:2369] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 21:14:08.949868 3144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: No bootstrap required, opened a new log
I20260810 21:14:08.950033 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.918108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.27.117.125:39851)
0810 21:14:08.918251 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.918253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.918266 (+ 13us) server_negotiation.cc:407] Connection header received
0810 21:14:08.918310 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.918311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.918333 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.918478 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.920602 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.922255 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.925850 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.926668 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.926831 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.937047 (+ 10216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.937049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.937054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.937071 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.942788 (+ 5717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.945013 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.945015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.945015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.945227 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.947345 (+ 2118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.947355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.947363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.947864 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.948377 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.949298 (+ 921us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.949637 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260810 21:14:08.957629 3144 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.958326 3144 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7647e7bb2ac4f69b98697d808ae49ae, State: Initialized, Role: FOLLOWER
I20260810 21:14:08.958770 3144 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.958241 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.916616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:41577 (local address 127.0.0.1:47708)
0810 21:14:08.916843 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:08.916864 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:08.916953 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:08.917238 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:08.917239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:08.917603 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:08.917683 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.917686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.919928 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.919930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.921537 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:08.921558 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.927620 (+ 6062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.927630 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:08.927815 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.937155 (+ 9340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:08.937171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:08.940151 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:08.947805 (+ 7654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.947815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.947815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.948888 (+ 1073us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.949855 (+ 967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:08.949857 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:08.949857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:08.950088 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:08.957256 (+ 7168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:08.957258 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:08.957356 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 21:14:08.957549 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 21:14:08.957861 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 21:14:08.960114 3144 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:08.961561 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:08.916932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.27.117.124:41577)
0810 21:14:08.917024 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:08.917030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:08.917078 (+ 48us) server_negotiation.cc:407] Connection header received
0810 21:14:08.917118 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:08.917119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:08.917140 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:08.917232 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:08.917963 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.919711 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.921792 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:08.922850 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:08.923177 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:08.940562 (+ 17385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:08.940574 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:08.940574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:08.940612 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:08.947470 (+ 6858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.949303 (+ 1833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.949304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.949304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.949526 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:08.956509 (+ 6983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:08.956515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:08.956516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:08.956928 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:08.957152 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:08.961173 (+ 4021us) server_negotiation.cc:299] Negotiation successful
0810 21:14:08.961480 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 21:14:08.961624 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:08.965270 2827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "53ed75047f1f461eadef81d4d3c45222"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41214
I20260810 21:14:08.967746 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222: Bootstrap starting.
I20260810 21:14:08.970674 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:08.972295 2874 log.cc:824] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:08.973531 3187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:08.973752 3187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:08.975363 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222: No bootstrap required, opened a new log
W20260810 21:14:08.975313 3205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:08.975565 3205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:14:08.968456 3014 master_runner.cc:186] Error getting master registration for 127.27.117.126:41009: OK, instance_id { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" instance_seqno: 1786396447947752 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 21:14:08.976155 3014 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:14:08.977792 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.978057 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ed75047f1f461eadef81d4d3c45222, State: Initialized, Role: FOLLOWER
I20260810 21:14:08.978264 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.978838 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:08.982313 2827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 21:14:08.982553 2827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:08.985575 2827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.987432 2952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53ed75047f1f461eadef81d4d3c45222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f7139fd3ace478e805a8e26e451b79d"
I20260810 21:14:08.987557 3212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:08.987954 3212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:08.987943 2952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [term 0 FOLLOWER]: Advancing to term 1
W20260810 21:14:08.989858 3233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:08.989964 3233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:14:08.990804 2952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ed75047f1f461eadef81d4d3c45222 in term 1.
I20260810 21:14:08.991204 2827 leader_election.cc:288] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [CANDIDATE]: Term 1 election: Requested vote from peers 4f7139fd3ace478e805a8e26e451b79d (127.27.117.125:39851), a7647e7bb2ac4f69b98697d808ae49ae (127.27.117.124:41577)
I20260810 21:14:08.991732 3092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53ed75047f1f461eadef81d4d3c45222" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a7647e7bb2ac4f69b98697d808ae49ae"
I20260810 21:14:08.991961 3092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:08.993831 3092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ed75047f1f461eadef81d4d3c45222 in term 1.
I20260810 21:14:08.994366 2765 leader_election.cc:302] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ed75047f1f461eadef81d4d3c45222, a7647e7bb2ac4f69b98697d808ae49ae; no voters:
I20260810 21:14:08.994674 3212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:08.995147 3212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [term 1 LEADER]: Becoming Leader. State: Replica: 53ed75047f1f461eadef81d4d3c45222, State: Running, Role: LEADER
I20260810 21:14:08.995244 3212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } }
I20260810 21:14:08.996930 3236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ed75047f1f461eadef81d4d3c45222. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:08.997527 3236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:09.002360 2732 master_runner.cc:263] No leader master found from master 53ed75047f1f461eadef81d4d3c45222
I20260810 21:14:09.002579 2732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:14:09.006585 3240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:14:09.007261 3240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:09.029644 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:09.000274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.27.117.125:39851)
0810 21:14:09.000376 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.000388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.007746 (+ 7358us) server_negotiation.cc:407] Connection header received
0810 21:14:09.009102 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.009103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.009144 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.009289 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.013044 (+ 3755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:09.014953 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:09.017687 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:09.018191 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:09.018300 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:09.021407 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:09.021414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:09.021414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:09.021434 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:09.025297 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:09.026725 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:09.026726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:09.026727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:09.026946 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:09.028069 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:09.028070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:09.028071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:09.028433 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:09.028603 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:09.028830 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 21:14:09.029254 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 21:14:09.037462 3092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [term 1 FOLLOWER]: Refusing update from remote peer 53ed75047f1f461eadef81d4d3c45222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:14:09.038901 2952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [term 1 FOLLOWER]: Refusing update from remote peer 53ed75047f1f461eadef81d4d3c45222: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:14:09.039646 3236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:14:09.040657 3236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:14:09.043737 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ed75047f1f461eadef81d4d3c45222. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:09.044148 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:09.048265 3187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ed75047f1f461eadef81d4d3c45222. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:09.048610 3187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:09.055152 3236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:09.055311 3236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:09.055507 3251 mvcc.cc:205] Tried to move back new op lower bound from 7317079855246934016 to 7317079855085838336. Current Snapshot: MvccSnapshot[applied={T|T < 7317079855246934016}]
I20260810 21:14:09.056434 3187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:09.056594 3187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:09.057003 3258 mvcc.cc:205] Tried to move back new op lower bound from 7317079855246934016 to 7317079855085838336. Current Snapshot: MvccSnapshot[applied={T|T < 7317079855246934016}]
I20260810 21:14:09.059561 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:09.005605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.27.117.126:41009)
0810 21:14:09.005831 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.005835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.013267 (+ 7432us) server_negotiation.cc:407] Connection header received
0810 21:14:09.013643 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.013645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.013711 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.013781 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.024945 (+ 11164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:09.027110 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:09.034232 (+ 7122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:09.035106 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:09.035393 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:09.044420 (+ 9027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:09.044430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:09.044431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:09.044468 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:09.051541 (+ 7073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:09.057235 (+ 5694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:09.057237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:09.057237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:09.057321 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:09.057721 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:09.057722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:09.057722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:09.058051 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:09.058197 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:09.059127 (+ 930us) server_negotiation.cc:299] Negotiation successful
0810 21:14:09.059381 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 21:14:09.061026 3212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:09.061439 3212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222 [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:09.062114 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53ed75047f1f461eadef81d4d3c45222" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ed75047f1f461eadef81d4d3c45222" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 41009 } } peers { permanent_uuid: "4f7139fd3ace478e805a8e26e451b79d" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 39851 } } peers { permanent_uuid: "a7647e7bb2ac4f69b98697d808ae49ae" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 41577 } } }
I20260810 21:14:09.062528 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:09.064258 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:09.006157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.27.117.124:41577)
0810 21:14:09.006218 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.006220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.016128 (+ 9908us) server_negotiation.cc:407] Connection header received
0810 21:14:09.016561 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.016562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.016584 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.016615 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.018397 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:09.019506 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:09.032827 (+ 13321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:09.033870 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:09.034124 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:09.052949 (+ 18825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:09.052951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:09.052952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:09.052976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:09.056260 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:09.058547 (+ 2287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:09.058548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:09.058549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:09.058647 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:09.061702 (+ 3055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:09.061703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:09.061704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:09.062062 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:09.062290 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:09.063572 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0810 21:14:09.063922 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 21:14:09.068936 3252 mvcc.cc:205] Tried to move back new op lower bound from 7317079855246934016 to 7317079855085838336. Current Snapshot: MvccSnapshot[applied={T|T < 7317079855246934016}]
I20260810 21:14:09.072088 3240 catalog_manager.cc:1347] Generated new cluster ID: bf9312a58c624b28922166965ed6814f
I20260810 21:14:09.072238 3240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:09.115754 3240 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:09.118263 3240 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:09.143617 3240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ed75047f1f461eadef81d4d3c45222: Generated new TSK 0
I20260810 21:14:09.143988 3240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 21:14:09.196296 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.191958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.27.117.125:39851)
0810 21:14:09.192133 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.192148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.194513 (+ 2365us) server_negotiation.cc:407] Connection header received
0810 21:14:09.194735 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.194737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.194780 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.194847 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.196165 (+ 1318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48918: BlockingRecv error: recv got EOF from 127.0.0.1:48918 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 21:14:09.198118 2877 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.193318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.27.117.126:41009)
0810 21:14:09.193386 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.193388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.197057 (+ 3669us) server_negotiation.cc:407] Connection header received
0810 21:14:09.197247 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.197248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.197269 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.197325 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.198010 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41240: BlockingRecv error: recv got EOF from 127.0.0.1:41240 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 21:14:09.201128 3156 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.193925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.27.117.124:41577)
0810 21:14:09.194093 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.194105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.198998 (+ 4893us) server_negotiation.cc:407] Connection header received
0810 21:14:09.199448 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.199449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.199530 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.199725 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.201019 (+ 1294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47722: BlockingRecv error: recv got EOF from 127.0.0.1:47722 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 21:14:09.211817 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.208445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.27.117.125:39851)
0810 21:14:09.208532 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.208534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.209439 (+ 905us) server_negotiation.cc:407] Connection header received
0810 21:14:09.209902 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.209912 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.209938 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.210131 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.211675 (+ 1544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48932: BlockingRecv error: recv got EOF from 127.0.0.1:48932 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 21:14:09.218453 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.216531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.27.117.125:39851)
0810 21:14:09.216609 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.216622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.217429 (+ 807us) server_negotiation.cc:407] Connection header received
0810 21:14:09.217630 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.217631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.217653 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.217710 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.218338 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48942: BlockingRecv error: recv got EOF from 127.0.0.1:48942 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 21:14:09.227849 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.224263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.27.117.125:39851)
0810 21:14:09.224498 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.224520 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.225473 (+ 953us) server_negotiation.cc:407] Connection header received
0810 21:14:09.225916 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.225926 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.225986 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.226159 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.227579 (+ 1420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48956: BlockingRecv error: recv got EOF from 127.0.0.1:48956 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
W20260810 21:14:09.236909 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.235313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.27.117.125:39851)
0810 21:14:09.235385 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.235387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.235813 (+ 426us) server_negotiation.cc:407] Connection header received
0810 21:14:09.236077 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.236078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.236099 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.236156 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.236746 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48958: BlockingRecv error: recv got EOF from 127.0.0.1:48958 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 21:14:09.251880 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.249027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.27.117.125:39851)
0810 21:14:09.249147 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.249159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.250247 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 21:14:09.250616 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.250619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.250641 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.250702 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.251719 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48964: BlockingRecv error: recv got EOF from 127.0.0.1:48964 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 21:14:09.297782 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.294015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.27.117.125:39851)
0810 21:14:09.294214 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.294228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.295772 (+ 1544us) server_negotiation.cc:407] Connection header received
0810 21:14:09.296422 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.296425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.296443 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.296654 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.297593 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48966: BlockingRecv error: recv got EOF from 127.0.0.1:48966 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260810 21:14:09.342952 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.336972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.27.117.125:39851)
0810 21:14:09.337017 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.337020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.342277 (+ 5257us) server_negotiation.cc:407] Connection header received
0810 21:14:09.342411 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.342413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.342428 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.342467 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.342889 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48980: BlockingRecv error: recv got EOF from 127.0.0.1:48980 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 21:14:09.409145 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.408295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.27.117.125:39851)
0810 21:14:09.408374 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.408376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.408537 (+ 161us) server_negotiation.cc:407] Connection header received
0810 21:14:09.408626 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.408627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.408640 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.408699 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.409082 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48996: BlockingRecv error: recv got EOF from 127.0.0.1:48996 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 21:14:09.538944 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.538261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.27.117.125:39851)
0810 21:14:09.538333 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.538335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.538448 (+ 113us) server_negotiation.cc:407] Connection header received
0810 21:14:09.538495 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.538496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.538505 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.538557 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.538890 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49012: BlockingRecv error: recv got EOF from 127.0.0.1:49012 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 21:14:09.796937 3009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:09.796252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.27.117.125:39851)
0810 21:14:09.796338 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:09.796341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:09.796351 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:14:09.796429 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:09.796430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:09.796444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:09.796522 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:09.796887 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49024: BlockingRecv error: recv got EOF from 127.0.0.1:49024 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 21:14:09.917793 3183 catalog_manager.cc:1259] Loaded cluster ID: bf9312a58c624b28922166965ed6814f
I20260810 21:14:09.917868 3183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: loading cluster ID for follower catalog manager: success
I20260810 21:14:09.918715 3183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: acquiring CA information for follower catalog manager: success
I20260810 21:14:09.919042 3183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f7139fd3ace478e805a8e26e451b79d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:14:09.976293 3205 catalog_manager.cc:1259] Loaded cluster ID: bf9312a58c624b28922166965ed6814f
I20260810 21:14:09.976393 3205 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: loading cluster ID for follower catalog manager: success
I20260810 21:14:09.977278 3205 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: acquiring CA information for follower catalog manager: success
I20260810 21:14:09.977595 3205 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7647e7bb2ac4f69b98697d808ae49ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:14:10.312731 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:10.310195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.27.117.125:39851)
0810 21:14:10.310329 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:10.310331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:10.310342 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:14:10.310408 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:10.310409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:10.310465 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:10.310544 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:10.310679 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.311179 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.311925 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.312125 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.312178 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:10.312323 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:10.312468 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:10.312572 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:10.312593 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 21:14:10.312612 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260810 21:14:10.316965 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:10.314652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.27.117.124:41577)
0810 21:14:10.314772 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:10.314774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:10.314788 (+ 14us) server_negotiation.cc:407] Connection header received
0810 21:14:10.314823 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:10.314823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:10.314858 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:10.314923 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:10.315002 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.315544 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.316205 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.316398 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.316446 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:10.316573 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:10.316657 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:10.316711 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:10.316818 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 21:14:10.316844 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":79,"threads_started":1}
I20260810 21:14:10.321502 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:10.318650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.27.117.124:41577)
0810 21:14:10.318689 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:10.318691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:10.319007 (+ 316us) server_negotiation.cc:407] Connection header received
0810 21:14:10.319081 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:10.319082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:10.319091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:10.319145 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:10.319241 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.319720 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.320750 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.320995 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.321070 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:10.321228 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:10.321256 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:10.321284 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:10.321380 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 21:14:10.321406 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:14:10.321604 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:10.318458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.27.117.125:39851)
0810 21:14:10.318496 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:10.318498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:10.318658 (+ 160us) server_negotiation.cc:407] Connection header received
0810 21:14:10.318750 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:10.318751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:10.318761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:10.318812 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:10.318897 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.319625 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.320715 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.321032 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.321089 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:10.321231 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:10.321261 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:10.321307 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:10.321506 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 21:14:10.321528 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:14:10.321789 3405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:10.318478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.27.117.126:41009)
0810 21:14:10.318672 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:10.318674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:10.318694 (+ 20us) server_negotiation.cc:407] Connection header received
0810 21:14:10.318735 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:10.318736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:10.318785 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:10.318864 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:10.319035 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.319781 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.320531 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:10.320847 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:10.320904 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:10.321130 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:10.321229 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:10.321293 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:10.321670 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 21:14:10.321702 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":57,"threads_started":1}
I20260810 21:14:10.323546 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2732
I20260810 21:14:10.343978 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 2884
I20260810 21:14:10.357300 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3014
2026-08-10T21:14:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3920 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 21:14:10 dist-test-slave-2kcd krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 21:14:10 dist-test-slave-2kcd krb5kdc[3410](info): set up 2 sockets
Aug 10 21:14:10 dist-test-slave-2kcd krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260810 21:14:11.622097 28117 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/test-user.keytab.
Aug 10 21:14:11 dist-test-slave-2kcd krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T21:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T21:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.117.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
Entry for principal kudu/127.27.117.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab.
I20260810 21:14:11.739346 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:36399
--webserver_interface=127.27.117.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:36399,127.27.117.125:35637,127.27.117.124:33081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 21:14:11.822860 3442 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 21:14:11.823103 3442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:11.823135 3442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:11.823181 3442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:11.824323 3442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:11.824391 3442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:11.824414 3442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:11.824435 3442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:11.824465 3442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:11.826011 3442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:36399,127.27.117.125:35637,127.27.117.124:33081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.126.keytab
--principal=kudu/127.27.117.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.126:36399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.117.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:11.826289 3442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:11.826522 3442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:11.831512 3454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:11.831512 3449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:11.831521 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:11.831948 3442 server_base.cc:1034] running on GCE node
I20260810 21:14:11.832211 3442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:11.832482 3442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:11.833638 3442 hybrid_clock.cc:648] HybridClock initialized: now 1786396451833610 us; error 49 us; skew 500 ppm
Aug 10 21:14:11 dist-test-slave-2kcd krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:11.840512 3442 init.cc:377] Logged in from keytab as kudu/127.27.117.126@KRBTEST.COM (short username kudu)
I20260810 21:14:11.841030 3442 webserver.cc:460] Webserver started at http://127.27.117.126:41961/ using document root <none> and password file <none>
I20260810 21:14:11.841293 3442 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:11.841339 3442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:11.841482 3442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:11.842584 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data/instance:
uuid: "1d14e9019aaf4dd0b10b9ffe335e155a"
format_stamp: "Formatted at 2026-08-10 21:14:11 on dist-test-slave-2kcd"
server_key: "e641344ce8a8f9785bcb2c3bd1e34846"
server_key_iv: "e0bf30248e8a05e89b980554da0e3758"
server_key_version: "encryptionkey@0"
I20260810 21:14:11.842953 3442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal/instance:
uuid: "1d14e9019aaf4dd0b10b9ffe335e155a"
format_stamp: "Formatted at 2026-08-10 21:14:11 on dist-test-slave-2kcd"
server_key: "e641344ce8a8f9785bcb2c3bd1e34846"
server_key_iv: "e0bf30248e8a05e89b980554da0e3758"
server_key_version: "encryptionkey@0"
I20260810 21:14:11.844327 3442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 21:14:11.845191 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:11.845374 3442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 21:14:11.845463 3442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal
uuid: "1d14e9019aaf4dd0b10b9ffe335e155a"
format_stamp: "Formatted at 2026-08-10 21:14:11 on dist-test-slave-2kcd"
server_key: "e641344ce8a8f9785bcb2c3bd1e34846"
server_key_iv: "e0bf30248e8a05e89b980554da0e3758"
server_key_version: "encryptionkey@0"
I20260810 21:14:11.845527 3442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:11.872028 3442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:11.874061 3442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:11.874178 3442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:11.878085 3442 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.126:36399
I20260810 21:14:11.878147 3553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.126:36399 every 8 connection(s)
I20260810 21:14:11.878499 3442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/data/info.pb
I20260810 21:14:11.879416 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:11.881053 3554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:11.884173 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3442
I20260810 21:14:11.884302 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-0/wal/instance
I20260810 21:14:11.884521 28117 external_mini_cluster.cc:1428] Setting key cc6b1e66c282d35271e10611fbc9626c
Aug 10 21:14:11 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:14:11.891314 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:11.881848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.27.117.126:36399)
0810 21:14:11.881992 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:11.881996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:11.882029 (+ 33us) server_negotiation.cc:407] Connection header received
0810 21:14:11.882556 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:11.882558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:11.882577 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:11.882684 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:11.883223 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:11.884046 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:11.885129 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:11.885446 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:11.885581 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:11.887924 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:11.887931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:11.887931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:11.887937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:11.889693 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:11.890090 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:11.890093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:11.890093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:11.890178 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:11.890412 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:11.890413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:11.890413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:11.890628 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:11.890717 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:11.890916 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 21:14:11.891065 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260810 21:14:11.891319 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:11.881692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36399 (local address 127.0.0.1:60160)
0810 21:14:11.881825 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:11.881856 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:11.882389 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:11.882691 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:11.882694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:11.882964 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:11.883118 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:11.883122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:11.884149 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:11.884150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:11.885042 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:11.885044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:11.885554 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:11.885555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:11.885621 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:11.886177 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:11.886187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:11.887823 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:11.889788 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:11.889795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:11.889797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:11.890002 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:11.890255 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:11.890256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:11.890256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:11.890344 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:11.890709 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:11.890714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:11.890776 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 21:14:11.890883 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 21:14:11.891128 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":94,"thread_start_us":72,"threads_started":1}
I20260810 21:14:11.892128 3554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:11.892503 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:11.892388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43192)
0810 21:14:11.892440 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:11.892483 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.27.117.125@KRBTEST.COM; defaulting to no policy
W20260810 21:14:11.892666 3470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
I20260810 21:14:11.893107 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:11.893019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43194)
0810 21:14:11.893066 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:11.893091 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 21:14:11.893266 3554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:35637: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab.
Entry for principal kudu/127.27.117.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab.
I20260810 21:14:11.909713 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:35637
--webserver_interface=127.27.117.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab
--principal=kudu/127.27.117.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:36399,127.27.117.125:35637,127.27.117.124:33081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:14:11.942488 3554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } attempt: 1
I20260810 21:14:11.943166 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:11.942972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43210)
0810 21:14:11.943050 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:11.943097 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 21:14:11.943573 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:11.943491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43212)
0810 21:14:11.943531 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:11.943556 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 21:14:11.943676 3554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:35637: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
W20260810 21:14:11.992848 3563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 21:14:11.993057 3563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:11.993074 3563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:11.993094 3563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:11.994170 3563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:11.994201 3563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:11.994216 3563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:11.994228 3563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:11.994240 3563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:11.995687 3563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:36399,127.27.117.125:35637,127.27.117.124:33081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.125.keytab
--principal=kudu/127.27.117.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.125:35637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.117.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:11.995882 3563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:11.996047 3563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:12.001029 3570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:12.001076 3575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:12.001004 3572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:12.001067 3563 server_base.cc:1034] running on GCE node
I20260810 21:14:12.001384 3563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:12.001677 3563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:12.002862 3563 hybrid_clock.cc:648] HybridClock initialized: now 1786396452002825 us; error 40 us; skew 500 ppm
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:12.005491 3563 init.cc:377] Logged in from keytab as kudu/127.27.117.125@KRBTEST.COM (short username kudu)
I20260810 21:14:12.005954 3563 webserver.cc:460] Webserver started at http://127.27.117.125:33433/ using document root <none> and password file <none>
I20260810 21:14:12.006151 3563 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:12.006206 3563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:12.006291 3563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:12.007313 3563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data/instance:
uuid: "fc9e37e2cd71473d88ab086f1ed52d26"
format_stamp: "Formatted at 2026-08-10 21:14:12 on dist-test-slave-2kcd"
server_key: "e9d5ff37d6b5566ff944de9751074434"
server_key_iv: "8d223fc17e9dd6e4a05e285d0de5118d"
server_key_version: "encryptionkey@0"
I20260810 21:14:12.007622 3563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal/instance:
uuid: "fc9e37e2cd71473d88ab086f1ed52d26"
format_stamp: "Formatted at 2026-08-10 21:14:12 on dist-test-slave-2kcd"
server_key: "e9d5ff37d6b5566ff944de9751074434"
server_key_iv: "8d223fc17e9dd6e4a05e285d0de5118d"
server_key_version: "encryptionkey@0"
I20260810 21:14:12.008968 3563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:14:12.009697 3584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:12.009874 3563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:14:12.009989 3563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal
uuid: "fc9e37e2cd71473d88ab086f1ed52d26"
format_stamp: "Formatted at 2026-08-10 21:14:12 on dist-test-slave-2kcd"
server_key: "e9d5ff37d6b5566ff944de9751074434"
server_key_iv: "8d223fc17e9dd6e4a05e285d0de5118d"
server_key_version: "encryptionkey@0"
I20260810 21:14:12.010066 3563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:12.011837 3554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } attempt: 2
I20260810 21:14:12.012248 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.012123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43226)
0810 21:14:12.012182 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.012220 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 21:14:12.012574 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.012476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43240)
0810 21:14:12.012521 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.012545 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 21:14:12.012663 3554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.125:35637: Network error: Client connection negotiation failed: client connection to 127.27.117.125:35637: connect: Connection refused (error 111)
I20260810 21:14:12.030660 3563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:12.032516 3563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:12.032676 3563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:12.036715 3563 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.125:35637
I20260810 21:14:12.036820 3680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.125:35637 every 8 connection(s)
I20260810 21:14:12.037145 3563 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/data/info.pb
I20260810 21:14:12.038014 3681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:12.039456 3681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:12.044705 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3563
I20260810 21:14:12.044850 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-1/wal/instance
I20260810 21:14:12.045044 28117 external_mini_cluster.cc:1428] Setting key c3ffd51dfc9f7c45d36ef4bd7b2d6e1e
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:14:12.048931 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.040135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.27.117.126:36399)
0810 21:14:12.040204 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.040206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.040361 (+ 155us) server_negotiation.cc:407] Connection header received
0810 21:14:12.040871 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.040872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.040884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.040941 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.041502 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.041968 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.042910 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.043122 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.043178 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.045674 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.045676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.045676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.045684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.047418 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.047826 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.047827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.047827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.047902 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.048189 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.048189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.048190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.048338 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.048394 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.048742 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.048863 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 21:14:12.049196 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.040074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36399 (local address 127.0.0.1:60170)
0810 21:14:12.040227 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.040267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.040726 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.040990 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.040992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.041227 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.041433 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.041437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.042046 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.042046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.042821 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.042823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.043203 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.043204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.043305 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.043771 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.043779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.045561 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.047507 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.047513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.047515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.047723 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.047969 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.047970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.047970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.048112 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.048408 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.048415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.048593 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.048719 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.048921 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260810 21:14:12.049794 3681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.117.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.117.124@KRBTEST.COM" created.
I20260810 21:14:12.057410 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.050076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43254)
0810 21:14:12.050133 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.050145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.050191 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.050422 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.050422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.050581 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.050644 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.050646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.051535 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.051537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.052373 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.052376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.052870 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.052871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.052953 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.053279 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.053284 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.054551 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.056338 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.056339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.056340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.056512 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.056733 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.056734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.056735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.056807 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.057078 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.057079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.057128 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.057187 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.057279 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 21:14:12.057503 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.050114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.27.117.125:35637)
0810 21:14:12.050266 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.050271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.050290 (+ 19us) server_negotiation.cc:407] Connection header received
0810 21:14:12.050311 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.050313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.050325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.050395 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.050723 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.051453 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.052462 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.052750 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.052872 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.054676 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.054684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.054684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.054691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.056227 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.056608 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.056611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.056611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.056667 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.056853 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.056854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.056855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.056980 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.057099 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.057224 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.057349 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260810 21:14:12.058027 3681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:12.058362 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.058244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34506)
0810 21:14:12.058291 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.058334 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 21:14:12.058502 3586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
I20260810 21:14:12.058887 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.058795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34516)
0810 21:14:12.058845 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.058868 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 21:14:12.059077 3681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:33081: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.117.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab.
Entry for principal kudu/127.27.117.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab.
I20260810 21:14:12.071923 28117 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksacWUR/build/release/bin/kudu
/tmp/dist-test-tasksacWUR/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:33081
--webserver_interface=127.27.117.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab
--principal=kudu/127.27.117.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.117.84:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.117.126:36399,127.27.117.125:35637,127.27.117.124:33081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 21:14:12.103796 3554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } attempt: 3
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:14:12.108294 3681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } attempt: 1
I20260810 21:14:12.108675 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.108529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34526)
0810 21:14:12.108603 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.108641 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 21:14:12.109097 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.108978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34536)
0810 21:14:12.109043 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.109070 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 21:14:12.109191 3681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:33081: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
I20260810 21:14:12.111155 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.104214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.27.117.125:35637)
0810 21:14:12.104285 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.104287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.104297 (+ 10us) server_negotiation.cc:407] Connection header received
0810 21:14:12.104324 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.104326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.104336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.104407 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.104717 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.105283 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.106002 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.106242 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.106323 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.108518 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.108520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.108520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.108527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.110126 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.110512 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.110512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.110513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.110561 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.110752 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.110753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.110753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.110846 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.110897 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.111048 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.111104 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 21:14:12.111220 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.104111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43258)
0810 21:14:12.104192 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.104207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.104267 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.104413 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.104414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.104570 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.104635 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.104637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.105391 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.105392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.105911 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.105913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.106320 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.106321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.106405 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.106860 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.106872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.108420 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.110210 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.110211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.110211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.110430 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.110623 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.110624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.110625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.110693 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.110906 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.110907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.110956 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.111032 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.111128 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 21:14:12.111752 3554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
I20260810 21:14:12.112074 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.111968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34540)
0810 21:14:12.112019 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.112051 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 21:14:12.112452 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.112353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34554)
0810 21:14:12.112403 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.112429 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 21:14:12.112576 3554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:33081: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
I20260810 21:14:12.143755 3554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } attempt: 1
I20260810 21:14:12.144201 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.144082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34560)
0810 21:14:12.144141 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.144175 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 21:14:12.144570 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.144480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34572)
0810 21:14:12.144527 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.144553 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 21:14:12.144671 3554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:33081: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
W20260810 21:14:12.154531 3692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 21:14:12.154744 3692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 21:14:12.154803 3692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 21:14:12.154845 3692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 21:14:12.155962 3692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 21:14:12.156024 3692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 21:14:12.156049 3692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 21:14:12.156081 3692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 21:14:12.156111 3692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 21:14:12.157649 3692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.117.84:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.117.126:36399,127.27.117.125:35637,127.27.117.124:33081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/krb5kdc/kudu_127.27.117.124.keytab
--principal=kudu/127.27.117.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.117.124:33081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.117.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 20:11:33 UTC on ccycloud.quasar-zccrtj.root.comops.site
build id 4579040
I20260810 21:14:12.157914 3692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 21:14:12.158144 3692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 21:14:12.162724 3698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:12.162763 3699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 21:14:12.163072 3707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 21:14:12.163282 3692 server_base.cc:1034] running on GCE node
I20260810 21:14:12.163530 3692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 21:14:12.163803 3692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 21:14:12.164963 3692 hybrid_clock.cc:648] HybridClock initialized: now 1786396452164954 us; error 43 us; skew 500 ppm
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 21:14:12.167318 3692 init.cc:377] Logged in from keytab as kudu/127.27.117.124@KRBTEST.COM (short username kudu)
I20260810 21:14:12.167757 3692 webserver.cc:460] Webserver started at http://127.27.117.124:40927/ using document root <none> and password file <none>
I20260810 21:14:12.167994 3692 fs_manager.cc:359] Metadata directory not provided
I20260810 21:14:12.168059 3692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 21:14:12.168223 3692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 21:14:12.169319 3692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data/instance:
uuid: "18baeed0f69141b58379c19375b001a6"
format_stamp: "Formatted at 2026-08-10 21:14:12 on dist-test-slave-2kcd"
server_key: "8b332344c71a3a7124867a3abaf89eb8"
server_key_iv: "787ae6863c875a416df01962c7c9463e"
server_key_version: "encryptionkey@0"
I20260810 21:14:12.169682 3692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal/instance:
uuid: "18baeed0f69141b58379c19375b001a6"
format_stamp: "Formatted at 2026-08-10 21:14:12 on dist-test-slave-2kcd"
server_key: "8b332344c71a3a7124867a3abaf89eb8"
server_key_iv: "787ae6863c875a416df01962c7c9463e"
server_key_version: "encryptionkey@0"
I20260810 21:14:12.171021 3692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 21:14:12.171696 3717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 21:14:12.171886 3692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 21:14:12.171985 3692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data,/tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal
uuid: "18baeed0f69141b58379c19375b001a6"
format_stamp: "Formatted at 2026-08-10 21:14:12 on dist-test-slave-2kcd"
server_key: "8b332344c71a3a7124867a3abaf89eb8"
server_key_iv: "787ae6863c875a416df01962c7c9463e"
server_key_version: "encryptionkey@0"
I20260810 21:14:12.172066 3692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 21:14:12.177322 3681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } attempt: 2
I20260810 21:14:12.177691 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.177572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34584)
0810 21:14:12.177631 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.177665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 21:14:12.178074 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.177991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34586)
0810 21:14:12.178034 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.178057 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 21:14:12.178194 3681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.117.124:33081: Network error: Client connection negotiation failed: client connection to 127.27.117.124:33081: connect: Connection refused (error 111)
I20260810 21:14:12.181790 3692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 21:14:12.183508 3692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 21:14:12.183609 3692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 21:14:12.187525 3692 rpc_server.cc:307] RPC server started. Bound to: 127.27.117.124:33081
I20260810 21:14:12.187554 3809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.117.124:33081 every 8 connection(s)
I20260810 21:14:12.187879 3692 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/data/info.pb
I20260810 21:14:12.188736 3811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 21:14:12.190204 3811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:14:12.196662 28117 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksacWUR/build/release/bin/kudu as pid 3692
I20260810 21:14:12.196789 28117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksacWUR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786396166878326-28117-0/minicluster-data/master-2/wal/instance
I20260810 21:14:12.197055 28117 external_mini_cluster.cc:1428] Setting key a119096eed30105b0eac501090d2b492
I20260810 21:14:12.198916 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.190835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.27.117.126:36399)
0810 21:14:12.190904 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.190907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.191117 (+ 210us) server_negotiation.cc:407] Connection header received
0810 21:14:12.191553 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.191554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.191563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.191613 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.192149 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.192589 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.193585 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.193762 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.193818 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.195761 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.195762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.195762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.195769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.197340 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.197861 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.197862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.197863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.197930 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.198228 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.198228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.198228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.198329 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.198392 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.198741 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.198827 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 21:14:12.198936 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.190815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.126:36399 (local address 127.0.0.1:60178)
0810 21:14:12.191004 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.191037 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.191477 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.191670 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.191673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.191909 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.192081 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.192084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.192690 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.192691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.193512 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.193514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.193838 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.193839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.193963 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.194365 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.194372 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.195674 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.197430 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.197437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.197439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.197780 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.198007 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.198009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.198009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.198153 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.198412 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.198420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.198582 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.198680 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.198792 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260810 21:14:12.199569 3811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
I20260810 21:14:12.206460 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.199885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.27.117.125:35637)
0810 21:14:12.199935 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.199937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.199946 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:14:12.199970 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.199971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.199982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.200043 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.200266 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.200869 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.201731 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.202110 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.202189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.203777 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.203778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.203778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.203785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.205256 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.205620 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.205621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.205621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.205670 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.205906 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.205907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.205907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.206055 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.206135 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.206291 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.206369 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 21:14:12.206490 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.199808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.125:35637 (local address 127.0.0.1:43266)
0810 21:14:12.199866 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.199876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.199919 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.200046 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.200046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.200153 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.200196 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.200197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.200939 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.200940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.201647 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.201650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.202193 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.202194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.202245 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.202609 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.202613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.203675 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.205331 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.205332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.205332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.205539 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.205738 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.205739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.205739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.205817 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.206144 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.206144 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.206189 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.206263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.206359 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 21:14:12.206715 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.197459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.27.117.124:33081)
0810 21:14:12.197589 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.197599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.197657 (+ 58us) server_negotiation.cc:407] Connection header received
0810 21:14:12.197724 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.197725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.197735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.197796 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.198550 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.199547 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.200545 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.200888 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.200975 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.203357 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.203365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.203365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.203372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.205132 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.205564 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.205566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.205566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.205660 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.205888 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.205889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.205890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.206070 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.206258 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.206301 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.206418 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260810 21:14:12.207007 3811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.124@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.126@KRBTEST.COM
I20260810 21:14:12.214054 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.207303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34608)
0810 21:14:12.207355 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.207366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.207403 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.207546 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.207548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.207698 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.207726 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.207728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.208505 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.208506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.209091 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.209093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.209551 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.209552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.209603 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.209981 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.209985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.211017 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.213078 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.213079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.213079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.213239 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.213427 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.213428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.213429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.213481 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.213736 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.213737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.213773 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.213857 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.213949 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 21:14:12.214120 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.207353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.27.117.124:33081)
0810 21:14:12.207384 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.207385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.207394 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:14:12.207470 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.207471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.207479 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.207522 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.207813 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.208425 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.209164 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.209475 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.209540 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.211102 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.211103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.211103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.211110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.212699 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.213317 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.213318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.213319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.213375 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.213554 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.213555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.213555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.213660 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.213731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.213882 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.213997 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":14}
I20260810 21:14:12.214974 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.207502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.27.117.126:36399)
0810 21:14:12.207540 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.207542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.207570 (+ 28us) server_negotiation.cc:407] Connection header received
0810 21:14:12.207591 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.207592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.207601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.207653 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.208177 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.208706 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.209550 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.209821 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.209904 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.212165 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.212166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.212167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.212174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.213598 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.213941 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.213942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.213943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.214001 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.214238 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.214239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.214240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.214394 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.214592 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.214785 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.214864 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:14:12.215600 3811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: Bootstrap starting.
I20260810 21:14:12.216529 3811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:12.216763 3811 log.cc:824] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:12.217528 3811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: No bootstrap required, opened a new log
I20260810 21:14:12.218930 3811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.219050 3811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18baeed0f69141b58379c19375b001a6, State: Initialized, Role: FOLLOWER
I20260810 21:14:12.219154 3811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.219482 3811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:12.219429 3818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.219544 3818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.219812 3554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } attempt: 2
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.117.125@KRBTEST.COM
W20260810 21:14:12.221529 3837 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:12.221626 3837 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396451, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.126@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:14:12.224571 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.215953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.27.117.125:35637)
0810 21:14:12.216847 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.216849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.216857 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:14:12.216878 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.216879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.216888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.216937 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.217383 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.217880 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.218515 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.218715 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.218791 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.221625 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.221626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.221627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.221634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.223264 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.223728 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.223730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.223730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.223788 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.224022 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.224022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.224023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.224159 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.224324 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.224384 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.224460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874}
I20260810 21:14:12.221762 3692 master_runner.cc:186] Error getting master registration for 127.27.117.126:36399: OK, instance_id { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" instance_seqno: 1786396451876989 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 21:14:12.224768 3692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:14:12.226507 28117 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 21:14:12.227571 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.220135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.27.117.124:33081)
0810 21:14:12.220174 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.220176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.220187 (+ 11us) server_negotiation.cc:407] Connection header received
0810 21:14:12.220211 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.220212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.220222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.220325 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.220645 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.221506 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.222359 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.222660 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.222748 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.224590 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.224591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.224591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.224597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.226356 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.226718 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.226719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.226719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.226772 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.227023 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.227024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.227024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.227149 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.227253 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.227425 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.227491 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 21:14:12.227622 3556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.220075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34616)
0810 21:14:12.220111 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.220122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.220164 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.220338 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.220339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.220511 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.220554 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.220556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.221608 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.221609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.222279 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.222282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.222752 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.222753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.222811 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.223187 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.223192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.224501 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.226431 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.226432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.226433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.226660 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.226845 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.226846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.226847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.226908 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.227248 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.227248 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.227292 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.227397 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.227491 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 21:14:12.227480 3504 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186
I20260810 21:14:12.229724 3554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a: Bootstrap starting.
I20260810 21:14:12.230300 3554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:12.230599 3554 log.cc:824] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:12.231303 3554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a: No bootstrap required, opened a new log
I20260810 21:14:12.232262 3554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.232396 3554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d14e9019aaf4dd0b10b9ffe335e155a, State: Initialized, Role: FOLLOWER
I20260810 21:14:12.232508 3554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.232756 3841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.232843 3554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:12.232869 3841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.232991 3504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 21:14:12.233067 3504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:12.233879 3504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.234117 3504 leader_election.cc:288] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [CANDIDATE]: Term 1 election: Requested vote from peers fc9e37e2cd71473d88ab086f1ed52d26 (127.27.117.125:35637), 18baeed0f69141b58379c19375b001a6 (127.27.117.124:33081)
W20260810 21:14:12.234620 3855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:12.234702 3855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:14:12.237133 3758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18baeed0f69141b58379c19375b001a6"
I20260810 21:14:12.237253 3758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:12.238155 3758 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d14e9019aaf4dd0b10b9ffe335e155a in term 1.
I20260810 21:14:12.235970 3442 master_runner.cc:186] Error getting master registration for 127.27.117.125:35637: OK, instance_id { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" instance_seqno: 1786396452035678 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 21:14:12.238360 3442 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 21:14:12.238400 3466 leader_election.cc:302] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18baeed0f69141b58379c19375b001a6, 1d14e9019aaf4dd0b10b9ffe335e155a; no voters:
I20260810 21:14:12.238524 3841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 21:14:12.238695 3841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [term 1 LEADER]: Becoming Leader. State: Replica: 1d14e9019aaf4dd0b10b9ffe335e155a, State: Running, Role: LEADER
I20260810 21:14:12.238782 3841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.239213 3857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d14e9019aaf4dd0b10b9ffe335e155a. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.239364 3857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:12.241142 3859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 21:14:12.241379 3859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 21:14:12.249711 3758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [term 1 FOLLOWER]: Refusing update from remote peer 1d14e9019aaf4dd0b10b9ffe335e155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:14:12.250607 3857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:14:12.252547 3818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d14e9019aaf4dd0b10b9ffe335e155a. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.252636 3818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.258226 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.240294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.27.117.125:35637)
0810 21:14:12.240332 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.240334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.241017 (+ 683us) server_negotiation.cc:407] Connection header received
0810 21:14:12.241142 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.241143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.241152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.241192 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.242559 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.243308 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.244813 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.245684 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.245732 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.248345 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.248346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.248346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.248354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.253063 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.256258 (+ 3195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.256273 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.256273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.256333 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.256913 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.256927 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.256928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.257067 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.257157 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.258040 (+ 883us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.258125 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:14:12.260537 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.240534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.27.117.126:36399)
0810 21:14:12.240569 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.240570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.242198 (+ 1628us) server_negotiation.cc:407] Connection header received
0810 21:14:12.242292 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.242293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.242302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.242330 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.243080 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.243761 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.245757 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.246059 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.246100 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.253955 (+ 7855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.253956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.253956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.253964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.255777 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.257669 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.257671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.257671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.257731 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.259203 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.259204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.259205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.259337 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.259427 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.260359 (+ 932us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.260437 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 21:14:12.263259 3857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.263340 3857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:12.263679 3881 mvcc.cc:205] Tried to move back new op lower bound from 7317079868412039168 to 7317079868370702336. Current Snapshot: MvccSnapshot[applied={T|T < 7317079868412039168}]
I20260810 21:14:12.264132 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.240738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.27.117.124:33081)
0810 21:14:12.240770 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.240772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.244737 (+ 3965us) server_negotiation.cc:407] Connection header received
0810 21:14:12.244995 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.244996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.245006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.245034 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.246428 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.247128 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.248969 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.249229 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.249277 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.260636 (+ 11359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.260637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.260637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.260644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.262510 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.263144 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.263145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.263145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.263199 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.263463 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.263464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.263464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.263585 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.263672 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.263973 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.264048 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 21:14:12.264289 3818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.264365 3818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.264830 3882 mvcc.cc:205] Tried to move back new op lower bound from 7317079868412039168 to 7317079868370702336. Current Snapshot: MvccSnapshot[applied={T|T < 7317079868412039168}]
I20260810 21:14:12.265542 3859 catalog_manager.cc:1347] Generated new cluster ID: ffd2d375f7ab44289aa80fba7d4203a5
I20260810 21:14:12.265655 3859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 21:14:12.269307 3681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } attempt: 3
Aug 10 21:14:12 dist-test-slave-2kcd krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786396452, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.117.125@KRBTEST.COM for kudu/127.27.117.124@KRBTEST.COM
I20260810 21:14:12.277437 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.269544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.117.124:33081 (local address 127.0.0.1:34628)
0810 21:14:12.269586 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 21:14:12.269596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 21:14:12.269630 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 21:14:12.269807 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 21:14:12.269809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 21:14:12.269978 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 21:14:12.270019 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.270021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.270916 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.270917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.271703 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 21:14:12.271705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.272181 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.272182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 21:14:12.272260 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.272748 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 21:14:12.272754 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 21:14:12.274539 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 21:14:12.276331 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.276332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.276333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.276539 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.276753 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 21:14:12.276754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 21:14:12.276754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 21:14:12.276858 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 21:14:12.277149 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 21:14:12.277150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 21:14:12.277200 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 21:14:12.277261 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 21:14:12.277346 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 21:14:12.277729 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:12.269593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.27.117.124:33081)
0810 21:14:12.269625 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.269627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.269635 (+ 8us) server_negotiation.cc:407] Connection header received
0810 21:14:12.269693 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.269693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.269702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.269758 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.270109 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.270835 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.271799 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:12.272099 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:12.272165 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:12.274701 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 21:14:12.274702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 21:14:12.274702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 21:14:12.274708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 21:14:12.276236 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.276627 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.276628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.276629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.276692 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 21:14:12.276937 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 21:14:12.276939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 21:14:12.276939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 21:14:12.277066 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 21:14:12.277124 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:12.277318 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 21:14:12.277394 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 21:14:12.279196 3681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: Bootstrap starting.
I20260810 21:14:12.279975 3681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: Neither blocks nor log segments found. Creating new log.
I20260810 21:14:12.280272 3681 log.cc:824] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: Log is configured to *not* fsync() on all Append() calls
I20260810 21:14:12.281421 3681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: No bootstrap required, opened a new log
I20260810 21:14:12.283383 3681 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.283519 3681 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9e37e2cd71473d88ab086f1ed52d26, State: Initialized, Role: FOLLOWER
I20260810 21:14:12.283598 3681 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } }
I20260810 21:14:12.283918 3681 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 21:14:12.283981 3632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 21:14:12.284878 3632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [term 1 FOLLOWER]: Refusing update from remote peer 1d14e9019aaf4dd0b10b9ffe335e155a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 21:14:12.285054 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc9e37e2cd71473d88ab086f1ed52d26"
I20260810 21:14:12.285764 3633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d14e9019aaf4dd0b10b9ffe335e155a in term 1.
I20260810 21:14:12.286150 3841 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 21:14:12.287448 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.287619 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.287720 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d14e9019aaf4dd0b10b9ffe335e155a. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.287815 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.287889 3892 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.287978 3892 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 21:14:12.290395 3917 mvcc.cc:205] Tried to move back new op lower bound from 7317079868412039168 to 7317079868370702336. Current Snapshot: MvccSnapshot[applied={T|T < 7317079868412039168}]
W20260810 21:14:12.291383 3922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 21:14:12.291441 3922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 21:14:12.292023 3841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d14e9019aaf4dd0b10b9ffe335e155a" member_type: VOTER last_known_addr { host: "127.27.117.126" port: 36399 } } peers { permanent_uuid: "fc9e37e2cd71473d88ab086f1ed52d26" member_type: VOTER last_known_addr { host: "127.27.117.125" port: 35637 } } peers { permanent_uuid: "18baeed0f69141b58379c19375b001a6" member_type: VOTER last_known_addr { host: "127.27.117.124" port: 33081 } } }
I20260810 21:14:12.292104 3841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a [sys.catalog]: This master's current role is: LEADER
I20260810 21:14:12.293340 3859 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 21:14:12.294056 3859 catalog_manager.cc:1504] Loading token signing keys...
I20260810 21:14:12.305791 3859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d14e9019aaf4dd0b10b9ffe335e155a: Generated new TSK 0
I20260810 21:14:12.305965 3859 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 21:14:12.335284 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.333961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.27.117.125:35637)
0810 21:14:12.334005 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.334007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.334631 (+ 624us) server_negotiation.cc:407] Connection header received
0810 21:14:12.334738 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.334740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.334751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.334782 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.335216 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43312: BlockingRecv error: recv got EOF from 127.0.0.1:43312 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 21:14:12.336181 3557 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.334357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.27.117.126:36399)
0810 21:14:12.334393 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.334395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.335696 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 21:14:12.335788 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.335789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.335798 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.335828 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.336122 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60198: BlockingRecv error: recv got EOF from 127.0.0.1:60198 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 21:14:12.337352 3816 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.336636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.27.117.124:33081)
0810 21:14:12.336675 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.336677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.336829 (+ 152us) server_negotiation.cc:407] Connection header received
0810 21:14:12.336950 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.336951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.336961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.336991 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.337295 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34630: BlockingRecv error: recv got EOF from 127.0.0.1:34630 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 21:14:12.339774 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.338968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.27.117.125:35637)
0810 21:14:12.339003 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.339005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.339240 (+ 235us) server_negotiation.cc:407] Connection header received
0810 21:14:12.339347 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.339348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.339366 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.339396 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.339731 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43328: BlockingRecv error: recv got EOF from 127.0.0.1:43328 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 21:14:12.342758 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.341962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.27.117.125:35637)
0810 21:14:12.341993 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.341995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.342253 (+ 258us) server_negotiation.cc:407] Connection header received
0810 21:14:12.342349 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.342350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.342360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.342388 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.342716 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43336: BlockingRecv error: recv got EOF from 127.0.0.1:43336 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 21:14:12.346776 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.345991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.27.117.125:35637)
0810 21:14:12.346025 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.346026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.346271 (+ 245us) server_negotiation.cc:407] Connection header received
0810 21:14:12.346368 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.346369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.346378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.346407 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.346737 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43342: BlockingRecv error: recv got EOF from 127.0.0.1:43342 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 21:14:12.352941 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.352050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.27.117.125:35637)
0810 21:14:12.352088 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.352090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.352339 (+ 249us) server_negotiation.cc:407] Connection header received
0810 21:14:12.352434 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.352435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.352445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.352474 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.352899 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43354: BlockingRecv error: recv got EOF from 127.0.0.1:43354 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 21:14:12.363095 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.362140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.27.117.125:35637)
0810 21:14:12.362189 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.362191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.362470 (+ 279us) server_negotiation.cc:407] Connection header received
0810 21:14:12.362642 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.362643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.362655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.362687 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.363034 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43370: BlockingRecv error: recv got EOF from 127.0.0.1:43370 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 21:14:12.381645 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.380531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.27.117.125:35637)
0810 21:14:12.380577 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.380580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.380973 (+ 393us) server_negotiation.cc:407] Connection header received
0810 21:14:12.381090 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.381091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.381104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.381137 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.381578 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43380: BlockingRecv error: recv got EOF from 127.0.0.1:43380 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 21:14:12.415875 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.415096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.27.117.125:35637)
0810 21:14:12.415148 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.415150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.415294 (+ 144us) server_negotiation.cc:407] Connection header received
0810 21:14:12.415369 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.415370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.415383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.415462 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.415814 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43394: BlockingRecv error: recv got EOF from 127.0.0.1:43394 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 21:14:12.481719 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.481007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.27.117.125:35637)
0810 21:14:12.481098 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.481100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.481218 (+ 118us) server_negotiation.cc:407] Connection header received
0810 21:14:12.481260 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.481261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.481272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.481346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.481647 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43402: BlockingRecv error: recv got EOF from 127.0.0.1:43402 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 21:14:12.611791 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.610958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.27.117.125:35637)
0810 21:14:12.611055 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.611057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.611163 (+ 106us) server_negotiation.cc:407] Connection header received
0810 21:14:12.611262 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.611263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.611276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.611368 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.611723 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43410: BlockingRecv error: recv got EOF from 127.0.0.1:43410 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 21:14:12.869765 3687 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 21:14:12.869060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.27.117.125:35637)
0810 21:14:12.869135 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:12.869137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:12.869167 (+ 30us) server_negotiation.cc:407] Connection header received
0810 21:14:12.869240 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:12.869241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:12.869251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:12.869294 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 21:14:12.869673 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43416: BlockingRecv error: recv got EOF from 127.0.0.1:43416 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 21:14:13.222143 3837 catalog_manager.cc:1259] Loaded cluster ID: ffd2d375f7ab44289aa80fba7d4203a5
I20260810 21:14:13.222234 3837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: loading cluster ID for follower catalog manager: success
I20260810 21:14:13.223042 3837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: acquiring CA information for follower catalog manager: success
I20260810 21:14:13.223294 3837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18baeed0f69141b58379c19375b001a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:14:13.291900 3922 catalog_manager.cc:1259] Loaded cluster ID: ffd2d375f7ab44289aa80fba7d4203a5
I20260810 21:14:13.291981 3922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: loading cluster ID for follower catalog manager: success
I20260810 21:14:13.292759 3922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: acquiring CA information for follower catalog manager: success
I20260810 21:14:13.293043 3922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc9e37e2cd71473d88ab086f1ed52d26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 21:14:13.385921 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:13.383066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.27.117.125:35637)
0810 21:14:13.383183 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:13.383185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:13.383200 (+ 15us) server_negotiation.cc:407] Connection header received
0810 21:14:13.383269 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:13.383270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:13.383334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:13.383421 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:13.383566 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.384064 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.385060 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.385271 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.385352 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:13.385559 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:13.385676 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:13.385768 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:13.385792 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 21:14:13.385815 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260810 21:14:13.390064 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:13.387807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.27.117.124:33081)
0810 21:14:13.387909 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:13.387910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:13.387919 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:14:13.387982 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:13.387983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:13.388029 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:13.388100 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:13.388186 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.388666 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.389405 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.389599 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.389654 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:13.389778 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:13.389875 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:13.389938 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:13.389979 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 21:14:13.390001 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260810 21:14:13.394621 4035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:13.391718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.27.117.125:35637)
0810 21:14:13.391761 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:13.391763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:13.391991 (+ 228us) server_negotiation.cc:407] Connection header received
0810 21:14:13.392117 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:13.392118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:13.392129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:13.392185 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:13.392303 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.393041 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.393822 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.394107 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.394182 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:13.394331 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:13.394365 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:13.394412 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:13.394522 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 21:14:13.394544 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:14:13.395135 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:13.391777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.27.117.124:33081)
0810 21:14:13.391820 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:13.391822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:13.391999 (+ 177us) server_negotiation.cc:407] Connection header received
0810 21:14:13.392047 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:13.392048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:13.392059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:13.392109 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:13.392255 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.393000 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.394031 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.394318 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.394378 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:13.394516 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:13.394544 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:13.394594 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:13.395028 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 21:14:13.395051 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 21:14:13.395668 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 21:14:13.391693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.27.117.126:36399)
0810 21:14:13.391848 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 21:14:13.391850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 21:14:13.391859 (+ 9us) server_negotiation.cc:407] Connection header received
0810 21:14:13.391996 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 21:14:13.391997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 21:14:13.392054 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 21:14:13.392128 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 21:14:13.392865 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.393590 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.394605 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 21:14:13.394891 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 21:14:13.394951 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 21:14:13.395164 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 21:14:13.395396 (+ 232us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 21:14:13.395481 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 21:14:13.395544 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 21:14:13.395566 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260810 21:14:13.397281 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3442
I20260810 21:14:13.423059 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3563
I20260810 21:14:13.432801 28117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksacWUR/build/release/bin/kudu with pid 3692
2026-08-10T21:14:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3082 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7002 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286558 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST